Nexa SDK is a comprehensive toolkit for supporting GGML and ONNX models. It supports text generation, image generation, vision-language models (VLM), Audio Language Model, auto-speech-recognition (ASR), and text-to-speech (TTS) capabilities.

A GUI Agent application based on UI-TARS(Vision-Lanuage Model) that allows you to control your computer using natural language.

An AI-powered file management tool that ensures privacy by organizing local texts, images. Using Llama3.2 3B and Llava v1.6 models with the Nexa SDK, it intuitively scans, restructures, and organizes files for quick, seamless access and easy retrieval.

Cambrian-1 is a family of multimodal LLMs with a vision-centric design.

Solve Visual Understanding with Reinforced VLMs

The code used to train and run inference with the ColPali architecture.

🚀 「大模型」1小时从0训练26M参数的视觉多模态VLM！🌏 Train a 26M-parameter VLM from scratch in just 1 hours!

[NeurIPS 2024] An official implementation of ShareGPT4Video: Improving Video Understanding and Generation with Better Captions

A curated list of 3D Vision papers relating to Robotics domain in the era of large models i.e. LLMs/VLMs, inspired by awesome-computer-vision, including papers, codes, and related websites

A novel Multimodal Large Language Model (MLLM) architecture, designed to structurally align visual and textual embeddings.

Eagle Family: Exploring Model Designs, Data Recipes and Training Strategies for Frontier-Class Multimodal LLMs

Parsing-free RAG supported by VLMs

AI-powered assistant to help you with your daily tasks, powered by Llama 3.2. It can recognize your voice, process natural language, and perform various actions based on your commands: summarizing text, rephasing sentences, answering questions, writing emails, and more.

Awesome-Jailbreak-on-LLMs is a collection of state-of-the-art, novel, exciting jailbreak methods on LLMs. It contains papers, codes, datasets, evaluations, and analyses.

The open source Meme Search Engine and Finder. Free and built to self-host locally with Python, Ruby, and Docker.

Use the reMarkable2 as an interface to vision-LLMs (ChatGPT, Claude, Gemini). Ghost in the machine!

An open-source implementation for training LLaVA-NeXT.

A hub for various industry-specific schemas to be used with VLMs.

JoyCaption is an image captioning Visual Language Model (VLM) being built from the ground up as a free, open, and uncensored model for the community to use in training Diffusion models.

RLAIF-V: Aligning MLLMs through Open-Source AI Feedback for Super GPT-4V Trustworthiness

EVE Series: Encoder-Free Vision-Language Models from BAAI

Phi-3.5 for Mac: Locally-run Vision and Language Models for Apple Silicon

Official Repository of paper VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding

The code for "TokenPacker: Efficient Visual Projector for Multimodal LLM".

Reliable Automation Agents at Scale

Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning

RAI is a multi-vendor agent framework for robotics, utilizing Langchain and ROS 2 tools to perform complex actions, defined scenarios, free interface execution, log summaries, voice interaction and more.

Seamlessly integrate state-of-the-art transformer models into robotics stacks

AUITestAgent is the first automatic, natural language-driven GUI testing tool for mobile apps, capable of fully automating the entire process of GUI interaction and function verification.

🔥[ICLR'25] LLaRA: Supercharging Robot Learning Data for Vision-Language Policy

This repo contains the code for "VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks" [ICLR25]

A toolbox for benchmarking trustworthiness of multimodal large language models (MultiTrust, NeurIPS 2024 Track Datasets and Benchmarks)

An open-source implementaion for fine-tuning Llama3.2-Vision series by Meta.

Benchmarking Agentic LLM and VLM Reasoning On Games

[NeurIPS 2024] Needle In A Multimodal Haystack (MM-NIAH): A comprehensive benchmark designed to systematically evaluate the capability of existing MLLMs to comprehend long multimodal documents.

A framework for standardizing evaluations of large foundation models, beyond single-score reporting and rankings.

Visualizing the attention of vision-language models

VLM driven tool that processes surveillance videos, extracts frames, and generates insightful annotations using a fine-tuned Florence-2 Vision-Language Model. Includes a Gradio-based interface for querying and analyzing video footage.

An image viewer and AI-assisted editing/captioning/masking tool that helps with curating datasets for generative AI models, finetunes and LoRA.

LLaVA-MORE: Enhancing Visual Instruction Tuning with LLaMA 3.1

VoCo-LLaMA: This repo is the official implementation of "VoCo-LLaMA: Towards Vision Compression with Large Language Models".

[NeurIPS 2024] CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs

Matryoshka Multimodal Models

The official code of the paper "Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Rate".

A minimal implementation of LLaVA-style VLM with interleaved image & text & video processing ability.