Nexa SDK is a comprehensive toolkit for supporting ONNX and GGML models. It supports text generation, image generation, vision-language models (VLM), auto-speech-recognition (ASR), and text-to-speech (TTS) capabilities.

Official repo for "Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models"

SoraWebui is an open-source Sora web client, enabling users to easily create videos from text with OpenAI's Sora model.

DeepSeek-VL: Towards Real-World Vision-Language Understanding

Cambrian-1 is a family of multimodal LLMs with a vision-centric design.

An AI-powered file management tool that ensures privacy by organizing local texts, images. Using Llama3.2 3B and Llava v1.6 models with the Nexa SDK, it intuitively scans, restructures, and organizes files for quick, seamless access and easy retrieval.

[NeurIPS 2024] An official implementation of ShareGPT4Video: Improving Video Understanding and Generation with Better Captions

MiniSora: A community aims to explore the implementation path and future development direction of Sora.

The code used to train and run inference with the ColPali architecture.

LLM Agent Framework in ComfyUI includes Omost,GPT-sovits, ChatTTS,GOT-OCR2.0, and FLUX prompt nodes,access to Feishu,discord,and adapts to all llms with similar openai/gemini interfaces, such as o1,ollama, grok, qwen, GLM, deepseek, moonshot,doubao. Adapted to local llms, vlm, gguf such as llama-3.2, Linkage neo4j KG, graphRAG / RAG / html 2 img

Sora AI Video Generator by Sora.FM

A family of lightweight multimodal models.

🔥🔥 LLaVA++: Extending LLaVA with Phi-3 and LLaMA-3 (LLaVA LLaMA-3, LLaVA Phi-3)

A Framework of Small-scale Large Multimodal Models

[ECCV2024] Grounded Multimodal Large Language Model with Localized Visual Tokenization

A curated list of 3D Vision papers relating to Robotics domain in the era of large models i.e. LLMs/VLMs, inspired by awesome-computer-vision, including papers, codes, and related websites

EAGLE: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders

A novel Multimodal Large Language Model (MLLM) architecture, designed to structurally align visual and textual embeddings.

MLX-VLM is a package for running Vision LLMs locally on your Mac using MLX.

Famous Vision Language Models and Their Architectures

AI-powered assistant to help you with your daily tasks, powered by Llama 3.2. It can recognize your voice, process natural language, and perform various actions based on your commands: summarizing text, rephasing sentences, answering questions, writing emails, and more.

Index your memes by their content and text, making them easily retrievable for your meme warfare pleasures. Find funny fast.

Parsing-free RAG supported by VLMs

An open-source implementation for training LLaVA-NeXT.

「大模型」3小时从0训练27M参数的视觉多模态VLM，个人显卡即可推理训练！

Awesome-Jailbreak-on-LLMs is a collection of state-of-the-art, novel, exciting jailbreak methods on LLMs. It contains papers, codes, datasets, evaluations, and analyses.

AI Device Template Featuring Whisper, TTS, Groq, Llama3, OpenAI and more

RLAIF-V: Aligning MLLMs through Open-Source AI Feedback for Super GPT-4V Trustworthiness

[CVPR 2024] Official PyTorch Code for "PromptKD: Unsupervised Prompt Distillation for Vision-Language Models"

Phi-3.5 for Mac: Locally-run Vision and Language Models for Apple Silicon

[NeurIPS'24 Spotlight] EVE: Encoder-Free Vision-Language Models

Sora AI Awesome List – Your go-to resource hub for all things Sora AI, OpenAI's groundbreaking model for crafting realistic scenes from text. Explore a curated collection of articles, videos, podcasts, and news about Sora's capabilities, advancements, and more.

The code for "TokenPacker: Efficient Visual Projector for Multimodal LLM".

The most powerful and modular Sora WebUI, api and backend with OpenAI's Sora Model. Collecting the highest quality prompts for Sora. using NextJs and Tailwind CSS

本项目开源基于NextJS的前端， 希望能够提供一个用于生成式AI的文字转视频， 尤其是电影从编剧到视频生成的Web前端平台参考。Everyone can become a director. The Nextjs front-end of an AI driven platform for automatic movie/video generation (form GPT script generation to text2video movie generation).这是一个免费试用AI视频创作平台，集成了基于GPT的视频剧本生成和视频生成功能。 我们的理想是让每个人都能成为导演，以最快的方式将日常中的任何创意转化为高质量的视频， 无论是电影、营销视频、还是自媒体视频。

Official code for Paper "Mantis: Multi-Image Instruction Tuning" (TMLR2024)

Seamlessly integrate state-of-the-art transformer models into robotics stacks

From scratch implementation of a vision language model in pure PyTorch

RAI is a multi-vendor agent framework for robotics, utilizing Langchain and ROS 2 tools to perform complex actions, defined scenarios, free interface execution, log summaries, voice interaction and more.

LLaRA: Large Language and Robotics Assistant

AUITestAgent is the first automatic, natural language-driven GUI testing tool for mobile apps, capable of fully automating the entire process of GUI interaction and function verification.

[ECCV 2024] Embodied Understanding of Driving Scenarios

PsyDI: Towards a Personalized and Progressively In-depth Chatbot for Psychological Measurements. (e.g. MBTI Measurement Agent)

Image Textualization: An Automatic Framework for Generating Rich and Detailed Image Descriptions (NeurIPS 2024)

JoyCaption is an image captioning Visual Language Model (VLM) being built from the ground up as a free, open, and uncensored model for the community to use in training Diffusion models.

Official implementation of CVPR'24 paper 'Toward Generalist Anomaly Detection via In-context Residual Learning with Few-shot Sample Prompts'.

A Survey on Vision-Language Geo-Foundation Models (VLGFMs)

basic google recaptcha solver using llava-v1.6-7b

Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning

[NeurIPS 2024] VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models

A toolbox for benchmarking trustworthiness of multimodal large language models (MultiTrust, NeurIPS 2024 Track Datasets and Benchmarks)

[NeurIPS 2024] Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?

This repo is a live list of papers on game playing and large multimodality model - "A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challenges".

[NeurIPS 2024] Needle In A Multimodal Haystack (MM-NIAH): A comprehensive benchmark designed to systematically evaluate the capability of existing MLLMs to comprehend long multimodal documents.

[ICML 2024] Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models

Official Repo of Graphist

A framework for standardizing evaluations of large foundation models, beyond single-score reporting and rankings.

LLaVA-MORE: Enhancing Visual Instruction Tuning with LLaMA 3.1

A minimal implementation of LLaVA-style VLM with interleaved image & text & video processing ability.

An open-source implementaion for fine-tuning Llama3.2-Vision series by Meta.

VoCo-LLaMA: This repo is the official implementation of "VoCo-LLaMA: Towards Vision Compression with Large Language Models".

Matryoshka Multimodal Models

The official code of the paper "Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Rate".

Visualizing the attention of vision-language models

[NeurIPS 2024] CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs

Unified Multi-modal IAA Baseline and Benchmark

[CVPR 2024] 🏡Know Your Neighbors: Improving Single-View Reconstruction via Spatial Vision-Language Reasoning

🌋👵🏻 Yo'LLaVA: Your Personalized Language and Vision Assistant

[CoRL 2024] VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Grounding

Multi-Aspect Vision Language Pretraining - CVPR2024

Self-host a ChatGPT-style web interface for Ollama 🦙

This repository contains the training, inference, evaluation code for SpeechLLM models and details about the model releases on huggingface.

Enhancing Large Vision Language Models with Self-Training on Image Comprehension.

[ICRA 2024] Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Models

[ECCV 2024] Elysium: Exploring Object-level Perception in Videos via MLLM

[NeurIPS'24 & ICMLW'24] CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models

Official implementation of paper "SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference" proposed by Peking University and UC Berkeley.

中文医学多模态大模型 Large Chinese Language-and-Vision Assistant for BioMedicine

A Rust library integrated with ONNXRuntime, providing a collection of Computer Vison and Vision-Language models.

FreeVA: Offline MLLM as Training-Free Video Assistant

[ECCV 2024] - Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillation

Evaluation framework for paper "VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?"

[ICML 2024] Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models.

[ICML 2024] "Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models"

Official implementation of our paper "Finetuned Multimodal Language Models are High-Quality Image-Text Data Filters".

[CVPR 2024 Highlight] ImageNet-D

Undergraduate Dissertation of Guilin University of Electronic Technology

Python scripts to use for captioning images with VLMs

The official implementation of the paper "MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity". The MMInstruct dataset includes 973K instructions from 24 domains and four instruction types.

AAPL: Adding Attributes to Prompt Learning for Vision-Language Models (CVPRw 2024)

CompBench evaluates the comparative reasoning of multimodal large language models (MLLMs) with 40K image pairs and questions across 8 dimensions of relative comparison: visual attribute, existence, state, emotion, temporality, spatiality, quantity, and quality. CompBench covers diverse visual domains, including animals, fashion, sports, and scenes.

A bug-free and improved implementation of LLaVA-UHD, based on the code from the official repo

Can multimodal LLM help visual place recognition?

[NeurIPS'24] Official implementation of paper "Unveiling the Tapestry of Consistency in Large Vision-Language Models".

GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI.

Code & Dataset for Paper: "Distill Visual Chart Reasoning Ability from LLMs to MLLMs"

[CVPR 2024] Situational Awareness Matters in 3D Vision Language Reasoning

[ICLR 2024 Spotlight 🔥 ] - [ Best Paper Award SoCal NLP 2023 🏆] - Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models

up-to-date curated list of state-of-the-art Large vision language models hallucinations research work, papers & resources

This is the official repository for Vista dataset - A Vietnamese multimodal dataset contains more than 700,000 samples of conversations and images