Top AI Project by Categories

A list of Top influential AI open source project listed by different categories. ( Data sourced from GitHub, updated automatically everyday.)
RankingsOrganization Account
Related Project
Project intro
Star count
1

dvlab-research

643 followers
-
MGM
Official repo for "Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models"
3.2K
2

NexaAI

459 followers
United States of America
nexa-sdk
Nexa SDK is a comprehensive toolkit for supporting ONNX and GGML models. It supports text generation, image generation, vision-language models (VLM), auto-speech-recognition (ASR), and text-to-speech (TTS) capabilities.
3.1K
3

X-PLUG

278 followers
-
MobileAgent
Mobile-Agent: The Powerful Mobile Device Operation Assistant Family
3.0K
4

SoraWebui

59 followers
-
SoraWebui
SoraWebui is an open-source Sora web client, enabling users to easily create videos from text with OpenAI's Sora model.
2.3K
5

deepseek-ai

2.1K followers
-
DeepSeek-VL
DeepSeek-VL: Towards Real-World Vision-Language Understanding
2.1K
6

cambrian-mllm

33 followers
-
cambrian
Cambrian-1 is a family of multimodal LLMs with a vision-centric design.
1.8K
7

QiuYannnn

34 followers
Los Angeles
Local-File-Organizer
An AI-powered file management tool that ensures privacy by organizing local texts, images. Using Llama3.2 3B and Llava v1.6 models with the Nexa SDK, it intuitively scans, restructures, and organizes files for quick, seamless access and easy retrieval.
1.6K
8

ShareGPT4Omni

20 followers
-
ShareGPT4Video
[NeurIPS 2024] An official implementation of ShareGPT4Video: Improving Video Understanding and Generation with Better Captions
1.3K
9

mini-sora

36 followers
-
minisora
MiniSora: A community aims to explore the implementation path and future development direction of Sora.
1.2K
10

illuin-tech

36 followers
Paris, France
colpali
The code used to train and run inference with the ColPali architecture.
1.0K
11

heshengtao

28 followers
-
comfyui_LLM_party
Dify in ComfyUI includes Omost,GPT-sovits, ChatTTS,GOT-OCR2.0, and FLUX prompt nodes,access to Feishu,discord,and adapts to all llms with similar openai/gemini interfaces, such as o1,ollama, grok, qwen, GLM, deepseek, moonshot,doubao. Adapted to local llms, vlm, gguf such as llama-3.2, Linkage neo4j KG, graphRAG / RAG / html 2 img
976
12

all-in-aigc

746 followers
-
sorafm
Sora AI Video Generator by Sora.FM
947
13

BAAI-DCAI

131 followers
-
Bunny
A family of lightweight multimodal models.
927
14

mbzuai-oryx

217 followers
-
LLaVA-pp
🔥🔥 LLaVA++: Extending LLaVA with Phi-3 and LLaMA-3 (LLaVA LLaMA-3, LLaVA Phi-3)
807
15

TinyLLaVA

6 followers
-
TinyLLaVA_Factory
A Framework of Small-scale Large Multimodal Models
635
16

FoundationVision

348 followers
-
Groma
[ECCV2024] Grounded Multimodal Large Language Model with Localized Visual Tokenization
557
17

zubair-irshad

239 followers
Silicon Valley, CA, USA
Awesome-Robotics-3D
A curated list of 3D Vision papers relating to Robotics domain in the era of large models i.e. LLMs/VLMs, inspired by awesome-computer-vision, including papers, codes, and related websites
543
18

NVlabs

5.9K followers
-
EAGLE
EAGLE: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders
535
19

AIDC-AI

28 followers
-
Ovis
A novel Multimodal Large Language Model (MLLM) architecture, designed to structurally align visual and textual embeddings.
507
20

Blaizzy

190 followers
Poland
mlx-vlm
MLX-VLM is a package for running Vision LLMs locally on your Mac using MLX.
468
21

nrl-ai

42 followers
-
llama-assistant
AI-powered assistant to help you with your daily tasks, powered by Llama 3.2. It can recognize your voice, process natural language, and perform various actions based on your commands: summarizing text, rephasing sentences, answering questions, writing emails, and more.
403
22

gokayfem

128 followers
Turkey
awesome-vlm-architectures
Famous Vision Language Models and Their Architectures
400
23

xiaoachen98

97 followers
-
Open-LLaVA-NeXT
An open-source implementation for training LLaVA-NeXT.
389
24

neonwatty

307 followers
-
meme_search
Index your memes by their content and text, making them easily retrievable for your meme warfare pleasures. Find funny fast.
357
25

jingyaogong

147 followers
China
minimind-v
「大模型」3小时从0训练27M参数的视觉多模态VLM,个人显卡即可推理训练!
335
26

OpenBMB

4.3K followers
-
VisRAG
Parsing-free RAG supported by VLMs
326
27

yueliu1999

259 followers
Singapore
Awesome-Jailbreak-on-LLMs
Awesome-Jailbreak-on-LLMs is a collection of state-of-the-art, novel, exciting jailbreak methods on LLMs. It contains papers, codes, datasets, evaluations, and analyses.
279
28

developersdigest

421 followers
-
ai-devices
AI Device Template Featuring Whisper, TTS, Groq, Llama3, OpenAI and more
278
29

JosefAlbers

24 followers
-
Phi-3-Vision-MLX
Phi-3.5 for Mac: Locally-run Vision and Language Models for Apple Silicon
234
30

RLHF-V

16 followers
-
RLAIF-V
RLAIF-V: Aligning MLLMs through Open-Source AI Feedback for Super GPT-4V Trustworthiness
230
31

zhengli97

84 followers
Hangzhou, China
PromptKD
[CVPR 2024] Official PyTorch Code for "PromptKD: Unsupervised Prompt Distillation for Vision-Language Models"
228
32

baaivision

537 followers
China
EVE
[NeurIPS'24 Spotlight] EVE: Encoder-Free Vision-Language Models
225
33
Awesome-Open-AI-Sora
Sora AI Awesome List – Your go-to resource hub for all things Sora AI, OpenAI's groundbreaking model for crafting realistic scenes from text. Explore a curated collection of articles, videos, podcasts, and news about Sora's capabilities, advancements, and more.
216
34

CircleRadon

63 followers
Hangzhou
TokenPacker
The code for "TokenPacker: Efficient Visual Projector for Multimodal LLM".
210
35

SoraFlows

1 followers
-
SoraFlows
The most powerful and modular Sora WebUI, api and backend with OpenAI's Sora Model. Collecting the highest quality prompts for Sora. using NextJs and Tailwind CSS
195
36

JiaqiLi404

7 followers
Hong Kong
IAmDirector-Text2Video-NextJS-Client
本项目开源基于NextJS的前端, 希望能够提供一个用于生成式AI的文字转视频, 尤其是电影从编剧到视频生成的Web前端平台参考。Everyone can become a director. The Nextjs front-end of an AI driven platform for automatic movie/video generation (form GPT script generation to text2video movie generation).这是一个免费试用AI视频创作平台,集成了基于GPT的视频剧本生成和视频生成功能。 我们的理想是让每个人都能成为导演,以最快的方式将日常中的任何创意转化为高质量的视频, 无论是电影、营销视频、还是自媒体视频。
189
37

TIGER-AI-Lab

160 followers
Canada
Mantis
Official code for Paper "Mantis: Multi-Image Instruction Tuning"
179
38

mbodiai

19 followers
United States of America
embodied-agents
Seamlessly integrate state-of-the-art transformer models into robotics stacks
162
39

AviSoori1x

93 followers
San Francisco
seemore
From scratch implementation of a vision language model in pure PyTorch
161
40

RobotecAI

152 followers
Poland
rai
RAI is a multi-vendor agent framework for robotics, utilizing Langchain and ROS 2 tools to perform complex actions, defined scenarios, free interface execution, log summaries, voice interaction and more.
154
41

LostXine

82 followers
Stony Brook, NY
LLaRA
LLaRA: Large Language and Robotics Assistant
153
42

bz-lab

1 followers
-
AUITestAgent
AUITestAgent is the first automatic, natural language-driven GUI testing tool for mobile apps, capable of fully automating the entire process of GUI interaction and function verification.
148
43

opendilab

1.3K followers
China
PsyDI
PsyDI: Towards a Personalized and Progressively In-depth Chatbot for Psychological Measurements. (e.g. MBTI Measurement Agent)
146
44

OpenDriveLab

1.5K followers
Hong Kong
ELM
[ECCV 2024] Embodied Understanding of Driving Scenarios
145
45

sterzhang

6 followers
-
image-textualization
Image Textualization: An Automatic Framework for Generating Rich and Detailed Image Descriptions (NeurIPS 2024)
140
46

mala-lab

55 followers
-
InCTRL
Official implementation of CVPR'24 paper 'Toward Generalist Anomaly Detection via In-context Residual Learning with Few-shot Sample Prompts'.
126
47

zytx121

127 followers
Singapore
Awesome-VLGFM
A Survey on Vision-Language Geo-Foundation Models (VLGFMs)
125
48

notune

12 followers
-
captcha-solver
basic google recaptcha solver using llava-v1.6-7b
120
49

fpgaminer

154 followers
-
joycaption
JoyCaption is an image captioning Visual Language Model (VLM) being built from the ground up as a free, open, and uncensored model for the community to use in training Diffusion models.
118
50

WangWenhao0716

67 followers
Sydney
VidProM
[NeurIPS 2024] VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models
109
51

ZebangCheng

6 followers
-
Emotion-LLaMA
Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning
106
52

xlang-ai

431 followers
-
Spider2-V
[NeurIPS 2024] Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?
106
53

thu-ml

698 followers
FIT Building, Tsinghua University, Beijing, China
MMTrustEval
A toolbox for benchmarking trustworthiness of multimodal large language models (MultiTrust, NeurIPS 2024 Track Datasets and Benchmarks)
101
54

BAAI-Agents

73 followers
-
GPA-LM
This repo is a live list of papers on game playing and large multimodality model - "A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challenges".
98
55

chs20

5 followers
-
RobustVLM
[ICML 2024] Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models
98
56

OpenGVLab

2.4K followers
-
MM-NIAH
[NeurIPS 2024] Needle In A Multimodal Haystack (MM-NIAH): A comprehensive benchmark designed to systematically evaluate the capability of existing MLLMs to comprehend long multimodal documents.
98
57

graphic-design-ai

5 followers
-
graphist
Official Repo of Graphist
97
58

thomas-yanxin

270 followers
Beijing China
KarmaVLM
🧘🏻‍♂️KarmaVLM (相生):A family of high efficiency and powerful visual language model.
84
59

aimagelab

121 followers
Modena, Italy
LLaVA-MORE
LLaVA-MORE: Enhancing Visual Instruction Tuning with LLaMA 3.1
84
60

kyegomez

1.5K followers
Palo Alto
MoE-Mamba
Implementation of MoE Mamba from the paper: "MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts" in Pytorch and Zeta
83
61

fangyuan-ksgk

34 followers
Singapore
Mini-LLaVA
A minimal implementation of LLaVA-style VLM with interleaved image & text & video processing ability.
83
62

microsoft

78.6K followers
Redmond, WA
eureka-ml-insights
A framework for standardizing evaluations of large foundation models, beyond single-score reporting and rankings.
82
63

Yxxxb

47 followers
Shenzhen
VoCo-LLaMA
VoCo-LLaMA: This repo is the official implementation of "VoCo-LLaMA: Towards Vision Compression with Large Language Models".
81
64

mu-cai

49 followers
Madison. WI
matryoshka-mm
Matryoshka Multimodal Models
81
65

princeton-nlp

1.2K followers
-
CharXiv
[NeurIPS 2024] CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs
75
66

shikiw

58 followers
-
Modality-Integration-Rate
The official code of the paper "Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Rate".
75
67

KwaiVGI

493 followers
-
Uniaa
Unified Multi-modal IAA Baseline and Benchmark
70
68

ruili3

50 followers
Zürich, Switzerland
Know-Your-Neighbors
[CVPR 2024] 🏡Know Your Neighbors: Improving Single-View Reconstruction via Spatial Vision-Language Reasoning
69
69

WisconsinAIVision

32 followers
-
YoLLaVA
🌋👵🏻 Yo'LLaVA: Your Personalized Language and Vision Assistant
66
70

zjysteven

52 followers
Durham, United States
VLM-Visualizer
Visualizing the attention of vision-language models
65
71

HieuPhan33

14 followers
Australia
CVPR2024_MAVL
Multi-Aspect Vision Language Pretraining - CVPR2024
62
72

fly-apps

285 followers
-
ollama-open-webui
Self-host a ChatGPT-style web interface for Ollama 🦙
60
73

skit-ai

42 followers
Bangalore, India
SpeechLLM
This repository contains the training, inference, evaluation code for SpeechLLM models and details about the model releases on huggingface.
58
74

yihedeng9

23 followers
-
STIC
Enhancing Large Vision Language Models with Self-Training on Image Comprehension.
57
75

richard-peng-xia

42 followers
Chapel Hill, NC, U.S.
CARES
[NeurIPS'24 & ICMLW'24] CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models
56
76

Hon-Wong

6 followers
-
Elysium
[ECCV 2024] Elysium: Exploring Object-level Perception in Videos via MLLM
56
77

FlyCole

30 followers
UK
Dream2Real
[ICRA 2024] Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Models
56
78

blib-la

17 followers
Germany
captain
Give your computer an AI Brain
54
79

robert-mcdermott

12 followers
Seattle
LLM-Image-Classification
Image Classification Testing with LLMs
51
80

2U1

29 followers
-
Llama3.2-Vision-Finetune
An open-source implementaion for fine-tuning Llama3.2-Vision series by Meta.
51
81

Gumpest

122 followers
Beijing
SparseVLMs
Official implementation of paper "SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference" proposed by Peking University and UC Berkeley.
50
82

OpenRobotLab

397 followers
-
VLM-Grounder
[CoRL 2024] VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Grounding
50
83

BUAADreamer

79 followers
Beijing
Chinese-LLaVA-Med
中文医学多模态大模型 Large Chinese Language-and-Vision Assistant for BioMedicine
49
84

whwu95

144 followers
-
FreeVA
FreeVA: Offline MLLM as Training-Free Video Assistant
48
85

jamjamjon

17 followers
Shanghai
usls
A Rust library integrated with ONNXRuntime, providing a collection of Computer Vison and Vision-Language models.
47
86

VisualWebBench

1 followers
-
VisualWebBench
Evaluation framework for paper "VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?"
46
87

miccunifi

38 followers
Firenze - Viale Morgagni 65 - Italia
KDPL
[ECCV 2024] - Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillation
44
88

ys-zong

17 followers
Edinburgh
VLGuard
[ICML 2024] Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models.
44
89

tmlr-group

92 followers
Hong Kong
WCA
[ICML 2024] "Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models"
43
90

Victorwz

96 followers
-
MLM_Filter
Official implementation of our paper "Finetuned Multimodal Language Models are High-Quality Image-Text Data Filters".
41
91

chenshuang-zhang

0 followers
-
imagenet_d
[CVPR 2024 Highlight] ImageNet-D
38
92

bonjour-npy

18 followers
-
UndergraduateDissertation
Undergraduate Dissertation of Guilin University of Electronic Technology
38
93

ProGamerGov

131 followers
Multiverse
VLM-Captioning-Tools
Python scripts to use for captioning images with VLMs
34
94

RaptorMai

69 followers
Columbus
CompBench
CompBench evaluates the comparative reasoning of multimodal large language models (MLLMs) with 40K image pairs and questions across 8 dimensions of relative comparison: visual attribute, existence, state, emotion, temporality, spatiality, quantity, and quality. CompBench covers diverse visual domains, including animals, fashion, sports, and scenes.
31
95

ParadoxZW

48 followers
Hangzhou, China
LLaVA-UHD-Better
A bug-free and improved implementation of LLaVA-UHD, based on the code from the official repo
31
96

yuecao0119

12 followers
-
MMInstruct
The official implementation of the paper "MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity". The MMInstruct dataset includes 973K instructions from 24 domains and four instruction types.
31
97

Gahyeonkim09

4 followers
Naju-si, South Korea
AAPL
AAPL: Adding Attributes to Prompt Learning for Vision-Language Models (CVPRw 2024)
30
98

ai4ce

183 followers
Brooklyn, NY, U.S.
LLM4VPR
Can multimodal LLM help visual place recognition?
30
99
ConBench
Official implementation of paper "Unveiling the Tapestry of Consistency in Large Vision-Language Models", NeurIPS 2024.
30
100

uni-medical

89 followers
-
GMAI-MMBench
GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI.
29