Top AI Project by Categories

Solve Visual Understanding with Reinforced VLMs

Nexa SDK is a comprehensive toolkit for supporting GGML and ONNX models. It supports text generation, image generation, vision-language models (VLM), Audio Language Model, auto-speech-recognition (ASR), and text-to-speech (TTS) capabilities.

SpatialLM: Large Language Model for Spatial Understanding

A GUI Agent application based on UI-TARS(Vision-Lanuage Model) that allows you to control your computer using natural language.

Skywork-R1V2:Multimodal Hybrid Reinforcement Learning for Reasoning (Best open-source multimodal reasoning model)

An AI-powered file management tool that ensures privacy by organizing local texts, images. Using Llama3.2 3B and Llava v1.6 models with the Nexa SDK, it intuitively scans, restructures, and organizes files for quick, seamless access and easy retrieval.

🚀 「大模型」1小时从0训练26M参数的视觉多模态VLM！🌏 Train a 26M-parameter VLM from scratch in just 1 hours!

This series will take you on a journey from the fundamentals of NLP and Computer Vision to the cutting edge of Vision-Language Models.

A curated list of 3D Vision papers relating to Robotics domain in the era of large models i.e. LLMs/VLMs, inspired by awesome-computer-vision, including papers, codes, and related websites

Parsing-free RAG supported by VLMs

AI-powered assistant to help you with your daily tasks, powered by Llama 3, DeepSeek R1, and many more models on HuggingFace.

A hub for various industry-specific schemas to be used with VLMs.

Use the reMarkable2 as an interface to vision-LLMs (ChatGPT, Claude, Gemini). Ghost in the machine!

Flame is an open-source multimodal AI system designed to translate UI design mockups into high-quality React code. It leverages vision-language modeling, automated data synthesis, and structured training workflows to bridge the gap between design and front-end development.

JoyCaption is an image captioning Visual Language Model (VLM) being built from the ground up as a free, open, and uncensored model for the community to use in training Diffusion models.

Reliable Automation Agents at Scale

A minimal codebase for finetuning large multimodal models, supporting llava-1.5/1.6, llava-interleave, llava-next-video, llava-onevision, llama-3.2-vision, qwen-vl, qwen2-vl, phi3-v etc.

Kolosal AI is an OpenSource and Lightweight alternative to LM Studio to run LLMs 100% offline on your device.

An open-source implementaion for fine-tuning Llama3.2-Vision series by Meta.

Code for ChatRex: Taming Multimodal LLM for Joint Perception and Understanding

This repo contains the code for "VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks" [ICLR25]

A CPU Realtime VLM in 500M. Surpassed Moondream2 and SmolVLM. Training from scratch with ease.

LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning

Benchmarking Agentic LLM and VLM Reasoning On Games

A framework for standardizing evaluations of large foundation models, beyond single-score reporting and rankings.

VLM driven tool that processes surveillance videos, extracts frames, and generates insightful annotations using a fine-tuned Florence-2 Vision-Language Model. Includes a Gradio-based interface for querying and analyzing video footage.

An image viewer and AI-assisted editing/captioning/masking tool that helps with curating datasets for generative AI models, finetunes and LoRA.

The PyVisionAI Official Repo

The official code of the paper "Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Rate".

Helpful DoggyBot: Open-World Object Fetching using Legged Robots and Vision-Language Models

A minimal implementation of LLaVA-style VLM with interleaved image & text & video processing ability.

[CoRL 2024] VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Grounding

TrustEval: A modular and extensible toolkit for comprehensive trust evaluation of generative foundation models (GenFMs)

Official implementation of paper "SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference".

探索 AI 的未來！聯發創新基地（MediaTek Research）首次開源的手機應用程式，讓你直接體驗我們最新研發的 AI 模型。透過手機，我們將 AI 科技帶入每個人的生活，完全離線運作，隱私更有保障。這是一個開源專案，我們熱烈歡迎開發者和愛好者一同參與，為 AI 技術發展貢獻力量。立即加入我們，一起打造更優質的 AI 體驗！

Enhance and modify high-quality compositions using real-time rendering and generative AI output without affecting a hero product asset.

Dingo: A Comprehensive Data Quality Evaluation Tool

Code & Dataset for Paper: "Distill Visual Chart Reasoning Ability from LLMs to MLLMs"

[CVPR'25] SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding

The official implementation of the paper "MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity". The MMInstruct dataset includes 973K instructions from 24 domains and four instruction types.

Human Demo Videos to Robot Action Plans

Emma-X: An Embodied Multimodal Action Model with Grounded Chain of Thought and Look-ahead Spatial Reasoning

GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI.

[ICLR 2025] Official implementation and benchmark evaluation repository of <PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding>

CompBench evaluates the comparative reasoning of multimodal large language models (MLLMs) with 40K image pairs and questions across 8 dimensions of relative comparison: visual attribute, existence, state, emotion, temporality, spatiality, quantity, and quality. CompBench covers diverse visual domains, including animals, fashion, sports, and scenes.

🎉 The code repository for "Parrot: Multilingual Visual Instruction Tuning" in PyTorch.

AIN - The First Arabic Inclusive Large Multimodal Model. It is a versatile bilingual LMM excelling in visual and contextual understanding across diverse domains.

[AAAI 2025]This repo contains evaluation code for the paper “UrBench: A Comprehensive Benchmark for Evaluating Large Multimodal Models in Multi-View Urban Scenarios”

An integration of Segment Anything Model, Molmo, and, Whisper to segment objects using voice and natural language.

[AAAI 2025] HiRED strategically drops visual tokens in the image encoding stage to improve inference efficiency for High-Resolution Vision-Language Models (e.g., LLaVA-Next) under a fixed token budget.

Document parser for RAG

READ THE README

A novel alignment framework that leverages image retrieval to mitigate hallucinations in Vision Language Models.

An AI-powered tool for summarizing YouTube videos by generating scene descriptions, translating them, and creating subtitled videos with text-to-speech narration

Blueprint for Ingesting massive volumes of live or archived videos and extract insights for summarization and interactive Q&A

WorldCuisines is an extensive multilingual and multicultural benchmark that spans 30 languages, covering a wide array of global cuisines.

vision language models finetuning notebooks & use cases (paligemma - florence .....)

Generalist Virtual Agents: A Survey on Autonomous Agents Across Digital Platforms

A curated list of Turkish AI models, datasets, papers

A Node.js CLI and library that uses OpenAI, Ollama, ZhipuAI, Google Gemini or Coze to write AI-generated image descriptions and/or tags to EXIF metadata by its content.

We introduce new approach, Token Reduction using CLIP Metric (TRIM), aimed at improving the efficiency of MLLMs without sacrificing their performance.

Probing the limitations of multimodal language models for chemistry and materials research

Computer Agent Arena Hub: Compare & Test AI Agents on Crowdsourced Real-World Computer Use Tasks

[ICLR 2025] - Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion

Can MLLMs Understand the Deep Implication Behind Chinese Images?

Securade.ai Sentinel - A monitoring and surveillance application that enables visual Q&A and video captioning for existing CCTV cameras.

The PyQt6 application using ColPali and OpenAI to show Efficient Document Retrieval with Vision Language Models

Transformers + Mambas + LSTMS All in One Model

REST API for computing cross-modal similarity between images and text using the ColPaLI vision-language model

Chitrarth: Bridging Vision and Language for a Billion People

Video Generation Benchmark

Get up and running with Llama 3.3, DeepSeek-R1, Phi-4, Gemma 2, and other large language models.

ide-cap-chan is a utility for batch image captioning with natural language using various VL models

DexGANGrasp: Dexterous Generative Adversarial Grasping Synthesis for Task-Oriented Manipulation - IEEE-RAS International Conference on Humanoid Robots (Humanoids) 2024 | DOI: 10.1109/Humanoids58906.2024.10769950

A focus on aligning room elements for better flow and space utilization.

Simple Gradio application integrated with Hugging Face Multimodals to support visual question answering chatbot and more features

TextSnap: Demo for Florence 2 model used in OCR tasks to extract and visualize text from images.

Reviews of papers on zero-shot anomaly detection using vision-Language models

Multimodal-VideoRAG: Using BridgeTower Embeddings and Large Vision Language Models

Intelligently Understanding Your Screenshots

軽量VLMのMiniCPM-V2.6のColaboratoryサンプル

In this we explore into visual Question Answering Using Gemini LLM and image was in URL or any other extension

SVLR: Scalable, Training-Free Visual Language Robotics: a modular multi-model framework for consumer-grade GPUs

Yet another set of LLM nodes for ComfyUI (for local/remote OpenAI-like APIs, multi-modal models supported)

This is the official repository for Cross-modality Information Check for Detecting Jailbreaking in Multimodal Large Language Models.

The Ultimate Guide to Effortlessly Creating AI Videos for Social Media Go From Text to Eye-Catching Videos in Just a Few Steps

ScaleDP is an Open-Source extension of Apache Spark for Document Processing

最新野卡wildcard虚拟信用卡使用指南：野卡是什么？如何开通野卡信用卡？如何为野卡充值和提现？

EgoNormia | Benchmarking Physical Social Norm Understanding in VLMs

This repository contains the code for SFT, RLHF, and DPO, designed for vision-based LLMs, including the LLaVA models and the LLaMA-3.2-vision models.

A small Vision language Model

This repository contains the implementation of the Vision-and-Language Transformer (ViLT) model fine-tuned for Visual Question Answering (VQA) tasks. The project is structured to be easy to set up and use, providing a streamlined approach for experimenting with different configurations and datasets.

🎬 OpenAI's Sora, a new text-to-video AI model, is set to launch later in 2024.

Fine-tuning Qwen2.5-VL for vision-language tasks | Optimized for Vision understanding | LoRA & PEFT support.

Minimalist implementation of PaliGemma 2 & PaliGemma VLM from scratch

Streamlit App Combining Vision, Language, and Audio AI Models

Sora是什么？如何使用Sora？Sora入口在哪？Sora订阅保姆级教程！

Open Source Video Understanding API and Large Vision Model Observability Platform.

Discord App for Interacting with local Ollama Models. Multiple Agents Supported!

Personalized Vision Language Models for Social Human-Robot Interactions