Nexa SDK is a comprehensive toolkit for supporting ONNX and GGML models. It supports text generation, image generation, vision-language models (VLM), auto-speech-recognition (ASR), and text-to-speech (TTS) capabilities.

Official repo for "Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models"

SoraWebui is an open-source Sora web client, enabling users to easily create videos from text with OpenAI's Sora model.

DeepSeek-VL: Towards Real-World Vision-Language Understanding

Cambrian-1 is a family of multimodal LLMs with a vision-centric design.

An AI-powered file management tool that ensures privacy by organizing local texts, images. Using Llama3.2 3B and Llava v1.6 models with the Nexa SDK, it intuitively scans, restructures, and organizes files for quick, seamless access and easy retrieval.

[NeurIPS 2024] An official implementation of ShareGPT4Video: Improving Video Understanding and Generation with Better Captions

MiniSora: A community aims to explore the implementation path and future development direction of Sora.

The code used to train and run inference with the ColPali architecture.

LLM Agent Framework in ComfyUI includes Omost,GPT-sovits, ChatTTS,GOT-OCR2.0, and FLUX prompt nodes,access to Feishu,discord,and adapts to all llms with similar openai/gemini interfaces, such as o1,ollama, grok, qwen, GLM, deepseek, moonshot,doubao. Adapted to local llms, vlm, gguf such as llama-3.2, Linkage neo4j KG, graphRAG / RAG / html 2 img

Sora AI Video Generator by Sora.FM

A family of lightweight multimodal models.

🔥🔥 LLaVA++: Extending LLaVA with Phi-3 and LLaMA-3 (LLaVA LLaMA-3, LLaVA Phi-3)

A Framework of Small-scale Large Multimodal Models

[ECCV2024] Grounded Multimodal Large Language Model with Localized Visual Tokenization

A curated list of 3D Vision papers relating to Robotics domain in the era of large models i.e. LLMs/VLMs, inspired by awesome-computer-vision, including papers, codes, and related websites

EAGLE: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders

A novel Multimodal Large Language Model (MLLM) architecture, designed to structurally align visual and textual embeddings.

MLX-VLM is a package for running Vision LLMs locally on your Mac using MLX.

Famous Vision Language Models and Their Architectures

AI-powered assistant to help you with your daily tasks, powered by Llama 3.2. It can recognize your voice, process natural language, and perform various actions based on your commands: summarizing text, rephasing sentences, answering questions, writing emails, and more.

Index your memes by their content and text, making them easily retrievable for your meme warfare pleasures. Find funny fast.

Parsing-free RAG supported by VLMs

An open-source implementation for training LLaVA-NeXT.

「大模型」3小时从0训练27M参数的视觉多模态VLM，个人显卡即可推理训练！

Awesome-Jailbreak-on-LLMs is a collection of state-of-the-art, novel, exciting jailbreak methods on LLMs. It contains papers, codes, datasets, evaluations, and analyses.

AI Device Template Featuring Whisper, TTS, Groq, Llama3, OpenAI and more

RLAIF-V: Aligning MLLMs through Open-Source AI Feedback for Super GPT-4V Trustworthiness

[CVPR 2024] Official PyTorch Code for "PromptKD: Unsupervised Prompt Distillation for Vision-Language Models"

Phi-3.5 for Mac: Locally-run Vision and Language Models for Apple Silicon

[NeurIPS'24 Spotlight] EVE: Encoder-Free Vision-Language Models

Sora AI Awesome List – Your go-to resource hub for all things Sora AI, OpenAI's groundbreaking model for crafting realistic scenes from text. Explore a curated collection of articles, videos, podcasts, and news about Sora's capabilities, advancements, and more.

The code for "TokenPacker: Efficient Visual Projector for Multimodal LLM".

The most powerful and modular Sora WebUI, api and backend with OpenAI's Sora Model. Collecting the highest quality prompts for Sora. using NextJs and Tailwind CSS

本项目开源基于NextJS的前端， 希望能够提供一个用于生成式AI的文字转视频， 尤其是电影从编剧到视频生成的Web前端平台参考。Everyone can become a director. The Nextjs front-end of an AI driven platform for automatic movie/video generation (form GPT script generation to text2video movie generation).这是一个免费试用AI视频创作平台，集成了基于GPT的视频剧本生成和视频生成功能。 我们的理想是让每个人都能成为导演，以最快的方式将日常中的任何创意转化为高质量的视频， 无论是电影、营销视频、还是自媒体视频。

Official code for Paper "Mantis: Multi-Image Instruction Tuning" (TMLR2024)

Seamlessly integrate state-of-the-art transformer models into robotics stacks

From scratch implementation of a vision language model in pure PyTorch

RAI is a multi-vendor agent framework for robotics, utilizing Langchain and ROS 2 tools to perform complex actions, defined scenarios, free interface execution, log summaries, voice interaction and more.

LLaRA: Large Language and Robotics Assistant

AUITestAgent is the first automatic, natural language-driven GUI testing tool for mobile apps, capable of fully automating the entire process of GUI interaction and function verification.

[ECCV 2024] Embodied Understanding of Driving Scenarios

PsyDI: Towards a Personalized and Progressively In-depth Chatbot for Psychological Measurements. (e.g. MBTI Measurement Agent)

Image Textualization: An Automatic Framework for Generating Rich and Detailed Image Descriptions (NeurIPS 2024)

JoyCaption is an image captioning Visual Language Model (VLM) being built from the ground up as a free, open, and uncensored model for the community to use in training Diffusion models.