Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM

Hanrong Ye, Chao-Han Huck Yang, Arushi Goel, Wei Huang, Ligeng Zhu, Yuanhang Su, Sean Lin, An-Chieh Cheng, Zhen Wan, Jinchuan Tian, Yuming Lou, Dong Yang, Zhijian Liu, Yukang Chen, Ambrish Dantrey, Ehsan Jahangiri, Sreyan Ghosh, Daguang Xu, Ehsan Hosseini-Asl, Danial Mohseni Taheri, Vidya Murali, Sifei Liu, Yao Lu, Oluwatobi Olabiyi, Yu-Chiang Frank Wang, Rafael Valle, Bryan Catanzaro, Andrew Tao, Song Han, Jan Kautz, Hongxu Yin, Pavlo Molchanov

TL;DR

OmniVinci introduces an open omni-modal LLM that unifies vision, audio, and language via three architectural innovations—OmniAlignNet for cross-modal alignment, Temporal Embedding Grouping for temporal organization, and Constrained Rotary Time Embedding for absolute timing—combined with a two-stage training regimen and a 24M-conversation data pipeline. The work demonstrates strong performance across omni-modal, audio, video, and image benchmarks, achieving state-of-the-art averages on omni-modal tasks while reducing training tokens to about 0.2T and enabling effective downstream applications in robotics, medical AI, and smart factories. Key findings include substantial cross-modal synergy in perception and reasoning, robust implicit/explicit omni-modal learning, and compelling RL-based advances for omni-modal reasoning. The approach emphasizes data-engineering and ablation-driven design to provide actionable insights for building frontier multi-modal LLMs with practical efficiency and broad utility.

Abstract

Advancing machine intelligence requires developing the ability to perceive across multiple modalities, much as humans sense the world. We introduce OmniVinci, an initiative to build a strong, open-source, omni-modal LLM. We carefully study the design choices across model architecture and data curation. For model architecture, we present three key innovations: (i) OmniAlignNet for strengthening alignment between vision and audio embeddings in a shared omni-modal latent space; (ii) Temporal Embedding Grouping for capturing relative temporal alignment between vision and audio signals; and (iii) Constrained Rotary Time Embedding for encoding absolute temporal information in omni-modal embeddings. We introduce a curation and synthesis pipeline that generates 24M single-modal and omni-modal conversations. We find that modalities reinforce one another in both perception and reasoning. Our model, OmniVinci, outperforms Qwen2.5-Omni with +19.05 on DailyOmni (cross-modal understanding), +1.7 on MMAR (audio), and +3.9 on Video-MME (vision), while using just 0.2T training tokens - a 6 times reduction compared to Qwen2.5-Omni's 1.2T. We finally demonstrate omni-modal advantages in downstream applications spanning robotics, medical AI, and smart factory.

OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM

TL;DR

OmniVinci introduces an open omni-modal LLM that unifies vision, audio, and language via three architectural innovations—OmniAlignNet for cross-modal alignment, Temporal Embedding Grouping for temporal organization, and Constrained Rotary Time Embedding for absolute timing—combined with a two-stage training regimen and a 24M-conversation data pipeline. The work demonstrates strong performance across omni-modal, audio, video, and image benchmarks, achieving state-of-the-art averages on omni-modal tasks while reducing training tokens to about 0.2T and enabling effective downstream applications in robotics, medical AI, and smart factories. Key findings include substantial cross-modal synergy in perception and reasoning, robust implicit/explicit omni-modal learning, and compelling RL-based advances for omni-modal reasoning. The approach emphasizes data-engineering and ablation-driven design to provide actionable insights for building frontier multi-modal LLMs with practical efficiency and broad utility.

Abstract

Advancing machine intelligence requires developing the ability to perceive across multiple modalities, much as humans sense the world. We introduce OmniVinci, an initiative to build a strong, open-source, omni-modal LLM. We carefully study the design choices across model architecture and data curation. For model architecture, we present three key innovations: (i) OmniAlignNet for strengthening alignment between vision and audio embeddings in a shared omni-modal latent space; (ii) Temporal Embedding Grouping for capturing relative temporal alignment between vision and audio signals; and (iii) Constrained Rotary Time Embedding for encoding absolute temporal information in omni-modal embeddings. We introduce a curation and synthesis pipeline that generates 24M single-modal and omni-modal conversations. We find that modalities reinforce one another in both perception and reasoning. Our model, OmniVinci, outperforms Qwen2.5-Omni with +19.05 on DailyOmni (cross-modal understanding), +1.7 on MMAR (audio), and +3.9 on Video-MME (vision), while using just 0.2T training tokens - a 6 times reduction compared to Qwen2.5-Omni's 1.2T. We finally demonstrate omni-modal advantages in downstream applications spanning robotics, medical AI, and smart factory.
Paper Structure (44 sections, 7 equations, 16 figures, 20 tables)

This paper contains 44 sections, 7 equations, 16 figures, 20 tables.

Table of Contents

  1. Introduction
  2. Model Architecture
  3. Omni-Modal Alignment Mechanism
  4. Training Strategy
  5. Modality-Specific Training
  6. Omni-Modal Joint Training
  7. Experiments
  8. Design Choice Ablation
  9. Visual-Audio Alignment Scheme
  10. Implicit and Explicit Learning
  11. Scaling and Evaluation
  12. Omni-Modal Benchmark
  13. Audio Benchmark
  14. Video Benchmark
  15. Image Benchmark
  16. ...and 29 more sections

Figures (16)

  • Figure 1: OmniVinci demonstrates strong performance across widely used omni-modal (+19.05 on Dailyomni), audio (+1.7 on MMAR), and vision (+3.9 on Video-MME) understanding benchmarks.
  • Figure 2: We introduce a foundation model for omni-modal understanding. Our model blends information from vision, audio, and text modalities into a unified omni-modal token sequence via the proposed omni-modal alignment mechanism.
  • Figure 3: Illustration of the proposed OmniAlignNet module.
  • Figure 4: Omni-modal captions generation pipeline. Video is segmented into 20-second clips. Visual and audio captions are generated independently for each segment, but lack cross-modal context and contain wrong understanding (modality-specific hallucination). A separate LLM performs cross-modal correction and summarization to create accurate omni-modal captions.
  • Figure 5: Pie chart of overall distribution of training data across modalities, showing proportions for image (36%), non-speech sound (21%), speech (17%), omni (15%), and video (11%).
  • ...and 11 more figures