Nexa SDK is a comprehensive toolkit for supporting GGML and ONNX models. It supports text generation, image generation, vision-language models (VLM), Audio Language Model, auto-speech-recognition (ASR), and text-to-speech (TTS) capabilities.

Cambrian-1 is a family of multimodal LLMs with a vision-centric design.

An AI-powered file management tool that ensures privacy by organizing local texts, images. Using Llama3.2 3B and Llava v1.6 models with the Nexa SDK, it intuitively scans, restructures, and organizes files for quick, seamless access and easy retrieval.

[NeurIPS 2024] An official implementation of ShareGPT4Video: Improving Video Understanding and Generation with Better Captions

The code used to train and run inference with the ColPali architecture.

LLM Agent Framework in ComfyUI includes Omost,GPT-sovits, ChatTTS,GOT-OCR2.0, and FLUX prompt nodes,access to Feishu,discord,and adapts to all llms with similar openai / aisuite interfaces, such as o1,ollama, gemini, grok, qwen, GLM, deepseek, moonshot,doubao. Adapted to local llms, vlm, gguf such as llama-3.2, Linkage graphRAG / RAG

🔥🔥 LLaVA++: Extending LLaVA with Phi-3 and LLaMA-3 (LLaVA LLaMA-3, LLaVA Phi-3)

MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on your Mac using MLX.

[ECCV2024] Grounded Multimodal Large Language Model with Localized Visual Tokenization

A curated list of 3D Vision papers relating to Robotics domain in the era of large models i.e. LLMs/VLMs, inspired by awesome-computer-vision, including papers, codes, and related websites

EAGLE: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders

A novel Multimodal Large Language Model (MLLM) architecture, designed to structurally align visual and textual embeddings.

AI-powered assistant to help you with your daily tasks, powered by Llama 3.2. It can recognize your voice, process natural language, and perform various actions based on your commands: summarizing text, rephasing sentences, answering questions, writing emails, and more.

Parsing-free RAG supported by VLMs

Index your memes by their content and text, making them easily retrievable for your meme warfare pleasures. Find funny fast.

Awesome-Jailbreak-on-LLMs is a collection of state-of-the-art, novel, exciting jailbreak methods on LLMs. It contains papers, codes, datasets, evaluations, and analyses.

An open-source implementation for training LLaVA-NeXT.

「大模型」3小时从0训练27M参数的视觉多模态VLM，个人显卡即可推理训练！

AI Device Template Featuring Whisper, TTS, Groq, Llama3, OpenAI and more

RLAIF-V: Aligning MLLMs through Open-Source AI Feedback for Super GPT-4V Trustworthiness

Phi-3.5 for Mac: Locally-run Vision and Language Models for Apple Silicon

[NeurIPS'24 Spotlight] EVE: Encoder-Free Vision-Language Models

The code for "TokenPacker: Efficient Visual Projector for Multimodal LLM".

Official code for Paper "Mantis: Multi-Image Instruction Tuning" (TMLR2024)

RAI is a multi-vendor agent framework for robotics, utilizing Langchain and ROS 2 tools to perform complex actions, defined scenarios, free interface execution, log summaries, voice interaction and more.

From scratch implementation of a vision language model in pure PyTorch

Seamlessly integrate state-of-the-art transformer models into robotics stacks

LLaRA: Large Language and Robotics Assistant

AUITestAgent is the first automatic, natural language-driven GUI testing tool for mobile apps, capable of fully automating the entire process of GUI interaction and function verification.

PsyDI: Towards a Personalized and Progressively In-depth Chatbot for Psychological Measurements. (e.g. MBTI Measurement Agent)

Image Textualization: An Automatic Framework for Generating Rich and Detailed Image Descriptions (NeurIPS 2024)

JoyCaption is an image captioning Visual Language Model (VLM) being built from the ground up as a free, open, and uncensored model for the community to use in training Diffusion models.

Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning

[NeurIPS 2024] Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?

A toolbox for benchmarking trustworthiness of multimodal large language models (MultiTrust, NeurIPS 2024 Track Datasets and Benchmarks)

[NeurIPS 2024] Needle In A Multimodal Haystack (MM-NIAH): A comprehensive benchmark designed to systematically evaluate the capability of existing MLLMs to comprehend long multimodal documents.

VLM driven tool that processes surveillance videos, extracts frames, and generates insightful annotations using a fine-tuned Florence-2 Vision-Language Model. Includes a Gradio-based interface for querying and analyzing video footage.

A framework for standardizing evaluations of large foundation models, beyond single-score reporting and rankings.

LLaVA-MORE: Enhancing Visual Instruction Tuning with LLaMA 3.1

An open-source implementaion for fine-tuning Llama3.2-Vision series by Meta.

The official code of the paper "Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Rate".

Matryoshka Multimodal Models

A minimal implementation of LLaVA-style VLM with interleaved image & text & video processing ability.

VoCo-LLaMA: This repo is the official implementation of "VoCo-LLaMA: Towards Vision Compression with Large Language Models".

Visualizing the attention of vision-language models

Benchmarking Agentic LLM and VLM Reasoning On Games

[NeurIPS 2024] CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs

🌋👵🏻 Yo'LLaVA: Your Personalized Language and Vision Assistant

[CoRL 2024] VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Grounding

This repository contains the training, inference, evaluation code for SpeechLLM models and details about the model releases on huggingface.

Enhancing Large Vision Language Models with Self-Training on Image Comprehension.

[NeurIPS'24 & ICMLW'24] CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models

Official implementation of paper "SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference" proposed by Peking University and UC Berkeley.

中文医学多模态大模型 Large Chinese Language-and-Vision Assistant for BioMedicine

FreeVA: Offline MLLM as Training-Free Video Assistant

[ECCV 2024] - Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillation

[ICML 2024] "Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models"

The official implementation of the paper "MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity". The MMInstruct dataset includes 973K instructions from 24 domains and four instruction types.

AAPL: Adding Attributes to Prompt Learning for Vision-Language Models (CVPRw 2024)

A bug-free and improved implementation of LLaVA-UHD, based on the code from the official repo

Code & Dataset for Paper: "Distill Visual Chart Reasoning Ability from LLMs to MLLMs"

Can multimodal LLM help visual place recognition?

CompBench evaluates the comparative reasoning of multimodal large language models (MLLMs) with 40K image pairs and questions across 8 dimensions of relative comparison: visual attribute, existence, state, emotion, temporality, spatiality, quantity, and quality. CompBench covers diverse visual domains, including animals, fashion, sports, and scenes.

GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI.

[NeurIPS'24] Official implementation of paper "Unveiling the Tapestry of Consistency in Large Vision-Language Models".

[CoRL2024] ThinkGrasp: A Vision-Language System for Strategic Part Grasping in Clutter. https://arxiv.org/abs/2407.11298

[CVPR 2024] Situational Awareness Matters in 3D Vision Language Reasoning

[ICLR 2024 Spotlight 🔥 ] - [ Best Paper Award SoCal NLP 2023 🏆] - Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models

Kling AI, Make Imagination Alive. This is a revolutionary text-to-video model like Sora. Kling AI WebUI is the open source project to integrate Kling AI Video Generation Model.

RET-CLIP: A Retinal Image Foundation Model Pre-trained with Clinical Diagnostic Reports

Emma-X: An Embodied Multimodal Action Model with Grounded Chain of Thought and Look-ahead Spatial Reasoning

Evaluation code and datasets for the ACL 2024 paper, VISTA: Visualized Text Embedding for Universal Multi-Modal Retrieval. The original code and model can be accessed at FlagEmbedding.

Code implementation for paper titled "HOI-Ref: Hand-Object Interaction Referral in Egocentric Vision"

Vistral-V: Visual Instruction Tuning for Vistral - Vietnamese Large Vision-Language Model.

Visual Instruction Tuning for Qwen2 Base Model

Document parser for RAG

Official implementation of MICCAI2024 paper "Evidential Concept Embedding Models: Towards Reliable Concept Explanations for Skin Disease Diagnosis"

Mark web pages for use with vision-language models

Llava, Ollama and Streamlit | Create POWERFUL Image Analyzer Chatbot for FREE - Windows & Mac

Enhance and modify high-quality compositions using real-time rendering and generative AI output without affecting a hero product asset.

A comprehensive collection of multilingual datasets and large language models, meticulously curated for evaluating and enhancing the performance of large language models across diverse languages and tasks.

Official code of the paper ORacle: Large Vision-Language Models for Knowledge-Guided Holistic OR Domain Modeling accepted at MICCAI 2024.

Scalable AI Inference Server for CPU and GPU with Node.js | Utilizes llama.cpp and parts of llamafile C/C++ core under the hood.

[AAAI 2025] HiRED strategically drops visual tokens in the image encoding stage to improve inference efficiency for High-Resolution Vision-Language Models (e.g., LLaVA-Next) under a fixed token budget.

Florence-2 is a novel vision foundation model with a unified, prompt-based representation for a variety of computer vision and vision-language tasks.

Official implementation of Leveraging Visual Tokens for Extended Text Contexts in Multi-Modal Learning

WorldCuisines is an extensive multilingual and multicultural benchmark that spans 30 languages, covering a wide array of global cuisines.

Brainhack TIL 2024: Team 12000SGDPLUSHIE

Okra, your all in one personal AI assistant

[ICML 2024] Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models

Generalist Virtual Agents: A Survey on Autonomous Agents Across Digital Platforms

Run SOTA Vision-Language Model Florence-2 on your data!

[NAACL 2024] Z-GMOT: Zero-shot Generic Multiple Object Tracking

Probing the limitations of multimodal language models for chemistry and materials research

Describe images by using LLMs

[MICCAI'24] Incorporating Clinical Guidelines through Adapting Multi-modal Large Language Model for Prostate Cancer PI-RADS Scoring

An integration of Segment Anything Model, Molmo, and, Whisper to segment objects using voice and natural language.

vision language models finetuning notebooks & use cases (paligemma - florence .....)

MICCAI 2024 - Disease-informed Adaptation of Vision-Language Models