SGLang is a fast serving framework for large language models and vision language models.

SUPIR aims at developing Practical Algorithms for Photo-Realistic Image Restoration In the Wild. Our new online demo is also released at suppixel.ai.

Mobile-Agent: The Powerful Mobile Device Operation Assistant Family

SoraWebui is an open-source Sora web client, enabling users to easily create videos from text with OpenAI's Sora model.

DeepSeek-VL: Towards Real-World Vision-Language Understanding

Cambrian-1 is a family of multimodal LLMs with a vision-centric design.

MiniSora: A community aims to explore the implementation path and future development direction of Sora.

An official implementation of ShareGPT4Video: Improving Video Understanding and Generation with Better Captions

Sora AI Video Generator by Sora.FM

A family of lightweight multimodal models.

Open-source evaluation toolkit of large vision-language models (LVLMs), support ~100 VLMs, 30+ benchmarks

🔥🔥 LLaVA++: Extending LLaVA with Phi-3 and LLaMA-3 (LLaVA LLaMA-3, LLaVA Phi-3)

[CVPR2024] The code for "Osprey: Pixel Understanding with Visual Instruction Tuning"

[CVPR 2024] Alpha-CLIP: A CLIP Model Focusing on Wherever You Want

A reading list for large models safety, security, and privacy (including Awesome LLM Security, Safety, etc.).

Grounded Multimodal Large Language Model with Localized Visual Tokenization

A Framework of Small-scale Large Multimodal Models

EAGLE: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders

AI Device Template Featuring Whisper, TTS, Groq, Llama3, OpenAI and more

[CVPR2024] ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts

Index your memes by their content and text, making them easily retrievable for your meme warfare pleasures. Find funny fast.

ScreenAgent: A Computer Control Agent Driven by Visual Language Large Model (IJCAI-24)

chatgpt和claude官网完整还原，包含其官网的全部功能。具有完善的用户体系和流量变现体系。

Sora AI Awesome List – Your go-to resource hub for all things Sora AI, OpenAI's groundbreaking model for crafting realistic scenes from text. Explore a curated collection of articles, videos, podcasts, and news about Sora's capabilities, advancements, and more.

[CVPR 2024 Highlight] OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation

Famous Vision Language Models and Their Architectures

The code used to train and run inference with the ColPali architecture.

The most powerful and modular Sora WebUI, api and backend with OpenAI's Sora Model. Collecting the highest quality prompts for Sora. using NextJs and Tailwind CSS

[CVPR'24] RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback

本项目开源基于NextJS的前端， 希望能够提供一个用于生成式AI的文字转视频， 尤其是电影从编剧到视频生成的Web前端平台参考。Everyone can become a director. The Nextjs front-end of an AI driven platform for automatic movie/video generation (form GPT script generation to text2video movie generation).这是一个免费试用AI视频创作平台，集成了基于GPT的视频剧本生成和视频生成功能。 我们的理想是让每个人都能成为导演，以最快的方式将日常中的任何创意转化为高质量的视频， 无论是电影、营销视频、还是自媒体视频。

Awesome LLM-related papers and repos on very comprehensive topics.

EVE: Encoder-Free Vision-Language Models from BAAI

MLX-VLM is a package for running Vision LLMs locally on your Mac using MLX.

Phi-3 for Mac: Locally-run Vision and Language Models for Apple Silicon

[CVPR 2024] Official PyTorch Code for "PromptKD: Unsupervised Prompt Distillation for Vision-Language Models"

An open-source implementation for training LLaVA-NeXT.

From scratch implementation of a vision language model in pure PyTorch

A Ruby gem for interacting with Ollama's API that allows you to run open source AI LLMs (Large Language Models) locally.

Seamlessly integrate state-of-the-art transformer models into robotics stacks

Evaluating text-to-image/video/3D models with VQAScore

Official code for Paper "Mantis: Multi-Image Instruction Tuning"

[ECCV 2024] Embodied Understanding of Driving Scenarios

Image Textualization: An Automatic Framework for Generating Rich and Detailed Image Descriptions

[CVPR 2024] Prompt Highlighter: Interactive Control for Multi-Modal LLMs

LLaRA: Large Language and Robotics Assistant

basic google recaptcha solver using llava-v1.6-7b

AUITestAgent is the first automatic, natural language-driven GUI testing tool for mobile apps, capable of fully automating the entire process of GUI interaction and function verification.

VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models

Official implementation of CVPR'24 paper 'Toward Generalist Anomaly Detection via In-context Residual Learning with Few-shot Sample Prompts'.

🧘🏻‍♂️KarmaVLM (相生)：A family of high efficiency and powerful visual language model.

A Survey on Vision-Language Geo-Foundation Models (VLGFMs)

Official Repo of Graphist

Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?

FreeGenius AI, an advanced AI assistant that can talk and take multi-step actions. Supports numerous open-source LLMs via Llama.cpp or Ollama or Groq Cloud API, with optional integration with AutoGen agents, OpenAI API, Google Gemini Pro and unlimited plugins.

[ECCV2024] Official code implementation of Merlin: Empowering Multimodal LLMs with Foresight Minds

[ICML 2024] Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models

Implementation of MoE Mamba from the paper: "MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts" in Pytorch and Zeta

RAI is a multi-vendor agent framework for robotics, utilizing Langchain and ROS 2 tools to perform complex actions, defined scenarios, free interface execution, log summaries, voice interaction and more.

PsyDI: Towards a Personalized and Progressively In-depth Chatbot for Psychological Measurements. (e.g. MBTI Measurement Agent)

A RLHF Infrastructure for Vision-Language Models

This is the official implementation of the paper "Needle In A Multimodal Haystack"

Unified Multi-modal IAA Baseline and Benchmark

[CVPR 2024] 🏡Know Your Neighbors: Improving Single-View Reconstruction via Spatial Vision-Language Reasoning

A toolbox for benchmarking trustworthiness of multimodal large language models (MultiTrust)

Matryoshka Multimodal Models

This repo is a live list of papers on game playing and large multimodality model - "A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challenges".

VoCo-LLaMA: This repo is the official implementation of "VoCo-LLaMA: Towards Vision Compression with Large Language Models".

Official PyTorch implementation and benchmark dataset for IGARSS 2024 ORAL paper: "Composed Image Retrieval for Remote Sensing"

A novel Multimodal Large Language Model (MLLM) architecture, designed to structurally align visual and textual embeddings.

M3DBench introduces a comprehensive 3D instruction-following dataset with support for interleaved multi-modal prompts. Furthermore, M3DBench provides a new benchmark to assess large models across 3D vision-centric tasks.

[ICRA 2024] Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Models

LLaVA-JP is a Japanese VLM trained by LLaVA method

Give your computer an AI Brain

CVPR2024: Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models

Enhancing Large Vision Language Models with Self-Training on Image Comprehension.

A curative list of 3D Vision papers relating to Robotics domain in the era of large models i.e. LLMs/VLMs, inspired by awesome-computer-vision, including papers, codes, and related websites

Self-host a ChatGPT-style web interface for Ollama 🦙

📖A small curated list of Awesome SD/DiT/ViT/Diffusion Inference with Distributed/Caching/Sampling: DistriFusion, PipeFusion, AsyncDiff, DeepCache, Block Caching etc.

Multi-Aspect Vision Language Pretraining - CVPR2024

CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs

Official implementation of our paper "Finetuned Multimodal Language Models are High-Quality Image-Text Data Filters".

LLaVA inference with multiple images at once for cross-image analysis.

FreeVA: Offline MLLM as Training-Free Video Assistant

Evaluation framework for paper "VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?"

Undergraduate Dissertation of Guilin University of Electronic Technology

[CVPR2024 Highlight] Official Code for "ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Object"

LLaVA-MORE: Enhancing Visual Instruction Tuning with LLaMA 3.1

[arXiv'24 & ICMLW'24] CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models

[ECCV 2024] Elysium: Exploring Object-level Perception in Videos via MLLM

This repository contains the training, inference, evaluation code for SpeechLLM models and details about the model releases on huggingface.

Image Classification Testing with LLMs

[ICML 2024] "Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models"

AAPL: Adding Attributes to Prompt Learning for Vision-Language Models (CVPRw 2024)

[ECCV 2024] - Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillation

中文医学多模态大模型 Large Chinese Language-and-Vision Assistant for BioMedicine

Official implementation of paper "Unveiling the Tapestry of Consistency in Large Vision-Language Models".

Python scripts to use for captioning images with VLMs

[ICML 2024] Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models.

A bug-free and improved implementation of LLaVA-UHD, based on the code from the official repo

Kling AI, Make Imagination Alive. This is a revolutionary text-to-video model like Sora. Kling AI WebUI is the open source project to integrate Kling AI Video Generation Model.