Solve Visual Understanding with Reinforced VLMs

Nexa SDK is a comprehensive toolkit for supporting GGML and ONNX models. It supports text generation, image generation, vision-language models (VLM), Audio Language Model, auto-speech-recognition (ASR), and text-to-speech (TTS) capabilities.

SpatialLM: Large Language Model for Spatial Understanding

A GUI Agent application based on UI-TARS(Vision-Lanuage Model) that allows you to control your computer using natural language.

An AI-powered file management tool that ensures privacy by organizing local texts, images. Using Llama3.2 3B and Llava v1.6 models with the Nexa SDK, it intuitively scans, restructures, and organizes files for quick, seamless access and easy retrieval.

Pioneering Multimodal Reasoning with CoT

🚀 「大模型」1小时从0训练26M参数的视觉多模态VLM！🌏 Train a 26M-parameter VLM from scratch in just 1 hours!

This series will take you on a journey from the fundamentals of NLP and Computer Vision to the cutting edge of Vision-Language Models.

A curated list of 3D Vision papers relating to Robotics domain in the era of large models i.e. LLMs/VLMs, inspired by awesome-computer-vision, including papers, codes, and related websites

Parsing-free RAG supported by VLMs

Eagle Family: Exploring Model Designs, Data Recipes and Training Strategies for Frontier-Class Multimodal LLMs

Awesome-Jailbreak-on-LLMs is a collection of state-of-the-art, novel, exciting jailbreak methods on LLMs. It contains papers, codes, datasets, evaluations, and analyses.

AI-powered assistant to help you with your daily tasks, powered by Llama 3, DeepSeek R1, and many more models on HuggingFace.

A hub for various industry-specific schemas to be used with VLMs.

Use the reMarkable2 as an interface to vision-LLMs (ChatGPT, Claude, Gemini). Ghost in the machine!

Flame is an open-source multimodal AI system designed to translate UI design mockups into high-quality React code. It leverages vision-language modeling, automated data synthesis, and structured training workflows to bridge the gap between design and front-end development.

JoyCaption is an image captioning Visual Language Model (VLM) being built from the ground up as a free, open, and uncensored model for the community to use in training Diffusion models.

Reliable Automation Agents at Scale

A minimal codebase for finetuning large multimodal models, supporting llava-1.5/1.6, llava-interleave, llava-next-video, llava-onevision, llama-3.2-vision, qwen-vl, qwen2-vl, phi3-v etc.

The code for "TokenPacker: Efficient Visual Projector for Multimodal LLM".

AUITestAgent is the first automatic, natural language-driven GUI testing tool for mobile apps, capable of fully automating the entire process of GUI interaction and function verification.

Kolosal AI is an OpenSource and Lightweight alternative to LM Studio to run LLMs 100% offline on your device.

Code for ChatRex: Taming Multimodal LLM for Joint Perception and Understanding

This repo contains the code for "VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks" [ICLR25]

DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception

A CPU Realtime VLM in 500M. Surpassed Moondream2 and SmolVLM. Training from scratch with ease.

An open-source implementaion for fine-tuning Llama3.2-Vision series by Meta.

LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning

Benchmarking Agentic LLM and VLM Reasoning On Games

A framework for standardizing evaluations of large foundation models, beyond single-score reporting and rankings.

VLM driven tool that processes surveillance videos, extracts frames, and generates insightful annotations using a fine-tuned Florence-2 Vision-Language Model. Includes a Gradio-based interface for querying and analyzing video footage.

An image viewer and AI-assisted editing/captioning/masking tool that helps with curating datasets for generative AI models, finetunes and LoRA.

The PyVisionAI Official Repo

The official code of the paper "Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Rate".

Helpful DoggyBot: Open-World Object Fetching using Legged Robots and Vision-Language Models

A minimal implementation of LLaVA-style VLM with interleaved image & text & video processing ability.

[CoRL 2024] VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Grounding

[NeurIPS 2024] Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding

TrustEval: A modular and extensible toolkit for comprehensive trust evaluation of generative foundation models (GenFMs)

Official implementation of paper "SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference".

探索 AI 的未來！聯發創新基地（MediaTek Research）首次開源的手機應用程式，讓你直接體驗我們最新研發的 AI 模型。透過手機，我們將 AI 科技帶入每個人的生活，完全離線運作，隱私更有保障。這是一個開源專案，我們熱烈歡迎開發者和愛好者一同參與，為 AI 技術發展貢獻力量。立即加入我們，一起打造更優質的 AI 體驗！

Enhance and modify high-quality compositions using real-time rendering and generative AI output without affecting a hero product asset.

Dingo: A Comprehensive Data Quality Evaluation Tool

[ECCV 2024] - Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillation

Code & Dataset for Paper: "Distill Visual Chart Reasoning Ability from LLMs to MLLMs"

[CVPR'25] SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding

The official implementation of the paper "MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity". The MMInstruct dataset includes 973K instructions from 24 domains and four instruction types.

Human Demo Videos to Robot Action Plans

Emma-X: An Embodied Multimodal Action Model with Grounded Chain of Thought and Look-ahead Spatial Reasoning

GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI.

[ICLR 2025] Official implementation and benchmark evaluation repository of <PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding>

CompBench evaluates the comparative reasoning of multimodal large language models (MLLMs) with 40K image pairs and questions across 8 dimensions of relative comparison: visual attribute, existence, state, emotion, temporality, spatiality, quantity, and quality. CompBench covers diverse visual domains, including animals, fashion, sports, and scenes.

🎉 The code repository for "Parrot: Multilingual Visual Instruction Tuning" in PyTorch.

AIN - The First Arabic Inclusive Large Multimodal Model. It is a versatile bilingual LMM excelling in visual and contextual understanding across diverse domains.

[AAAI 2025]This repo contains evaluation code for the paper “UrBench: A Comprehensive Benchmark for Evaluating Large Multimodal Models in Multi-View Urban Scenarios”

Evaluation code and datasets for the ACL 2024 paper, VISTA: Visualized Text Embedding for Universal Multi-Modal Retrieval. The original code and model can be accessed at FlagEmbedding.

Florence-2 is a novel vision foundation model with a unified, prompt-based representation for a variety of computer vision and vision-language tasks.

An integration of Segment Anything Model, Molmo, and, Whisper to segment objects using voice and natural language.

[AAAI 2025] HiRED strategically drops visual tokens in the image encoding stage to improve inference efficiency for High-Resolution Vision-Language Models (e.g., LLaVA-Next) under a fixed token budget.

Code for GeoX: Geometric Problem Solving Through Unified Formalized Vision-Language Pre-training

Document parser for RAG

READ THE README

A novel alignment framework that leverages image retrieval to mitigate hallucinations in Vision Language Models.

An AI-powered tool for summarizing YouTube videos by generating scene descriptions, translating them, and creating subtitled videos with text-to-speech narration

Blueprint for Ingesting massive volumes of live or archived videos and extract insights for summarization and interactive Q&A

WorldCuisines is an extensive multilingual and multicultural benchmark that spans 30 languages, covering a wide array of global cuisines.

vision language models finetuning notebooks & use cases (paligemma - florence .....)

Generalist Virtual Agents: A Survey on Autonomous Agents Across Digital Platforms

A curated list of Turkish AI models, datasets, papers

A Node.js CLI and library that uses OpenAI, Ollama, ZhipuAI, Google Gemini or Coze to write AI-generated image descriptions and/or tags to EXIF metadata by its content.

[MICCAI'24] Incorporating Clinical Guidelines through Adapting Multi-modal Large Language Model for Prostate Cancer PI-RADS Scoring

We introduce new approach, Token Reduction using CLIP Metric (TRIM), aimed at improving the efficiency of MLLMs without sacrificing their performance.

Probing the limitations of multimodal language models for chemistry and materials research

Computer Agent Arena Hub: Compare & Test AI Agents on Crowdsourced Real-World Computer Use Tasks

Run SOTA Vision-Language Model Florence-2 on your data!

Can MLLMs Understand the Deep Implication Behind Chinese Images?

Securade.ai Sentinel - A monitoring and surveillance application that enables visual Q&A and video captioning for existing CCTV cameras.

The PyQt6 application using ColPali and OpenAI to show Efficient Document Retrieval with Vision Language Models

Transformers + Mambas + LSTMS All in One Model

REST API for computing cross-modal similarity between images and text using the ColPaLI vision-language model

Chitrarth: Bridging Vision and Language for a Billion People

Video Generation Benchmark

Get up and running with Llama 3.3, DeepSeek-R1, Phi-4, Gemma 2, and other large language models.

ide-cap-chan is a utility for batch image captioning with natural language using various VL models

VELOCITI Benchmark Evaluation and Visualisation Code

This SDK generates datasets for training Video LLMs from youtube videos.

DexGANGrasp: Dexterous Generative Adversarial Grasping Synthesis for Task-Oriented Manipulation - IEEE-RAS International Conference on Humanoid Robots (Humanoids) 2024 | DOI: 10.1109/Humanoids58906.2024.10769950

A focus on aligning room elements for better flow and space utilization.

Simple Gradio application integrated with Hugging Face Multimodals to support visual question answering chatbot and more features

TextSnap: Demo for Florence 2 model used in OCR tasks to extract and visualize text from images.

Reviews of papers on zero-shot anomaly detection using vision-Language models

Multimodal-VideoRAG: Using BridgeTower Embeddings and Large Vision Language Models

Image analysis with Ollama (AI models) from within Home Assistant

Intelligently Understanding Your Screenshots

軽量VLMのMiniCPM-V2.6のColaboratoryサンプル

In this we explore into visual Question Answering Using Gemini LLM and image was in URL or any other extension

SVLR: Scalable, Training-Free Visual Language Robotics: a modular multi-model framework for consumer-grade GPUs

Yet another set of LLM nodes for ComfyUI (for local/remote OpenAI-like APIs, multi-modal models supported)

This is the official repository for Cross-modality Information Check for Detecting Jailbreaking in Multimodal Large Language Models.

The Ultimate Guide to Effortlessly Creating AI Videos for Social Media Go From Text to Eye-Catching Videos in Just a Few Steps