Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation

Chetwin Low, Weimin Wang, Calder Katyal

TL;DR

Ovi introduces a unified audio–video generator with symmetric twin backbones that fuse timing and semantics via bidirectional cross-attention and scaled RoPE. The model is trained in two stages—foundational audio pretraining followed by AV fusion—using a single shared text prompt to align modalities without post hoc handling. Empirical results show competitive audio and video quality and strong synchronization against baselines, demonstrating the practicality of end-to-end joint AV generation. The approach advances scalable, movie-grade AV synthesis and provides a template for future unified multimodal generation systems.

Abstract

Audio-video generation has often relied on complex multi-stage architectures or sequential synthesis of sound and visuals. We introduce Ovi, a unified paradigm for audio-video generation that models the two modalities as a single generative process. By using blockwise cross-modal fusion of twin-DiT modules, Ovi achieves natural synchronization and removes the need for separate pipelines or post hoc alignment. To facilitate fine-grained multimodal fusion modeling, we initialize an audio tower with an architecture identical to that of a strong pretrained video model. Trained from scratch on hundreds of thousands of hours of raw audio, the audio tower learns to generate realistic sound effects, as well as speech that conveys rich speaker identity and emotion. Fusion is obtained by jointly training the identical video and audio towers via blockwise exchange of timing (via scaled-RoPE embeddings) and semantics (through bidirectional cross-attention) on a vast video corpus. Our model enables cinematic storytelling with natural speech and accurate, context-matched sound effects, producing movie-grade video clips. All the demos, code and model weights are published at https://aaxwaz.github.io/Ovi

Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation

TL;DR

Ovi introduces a unified audio–video generator with symmetric twin backbones that fuse timing and semantics via bidirectional cross-attention and scaled RoPE. The model is trained in two stages—foundational audio pretraining followed by AV fusion—using a single shared text prompt to align modalities without post hoc handling. Empirical results show competitive audio and video quality and strong synchronization against baselines, demonstrating the practicality of end-to-end joint AV generation. The approach advances scalable, movie-grade AV synthesis and provides a template for future unified multimodal generation systems.

Abstract

Audio-video generation has often relied on complex multi-stage architectures or sequential synthesis of sound and visuals. We introduce Ovi, a unified paradigm for audio-video generation that models the two modalities as a single generative process. By using blockwise cross-modal fusion of twin-DiT modules, Ovi achieves natural synchronization and removes the need for separate pipelines or post hoc alignment. To facilitate fine-grained multimodal fusion modeling, we initialize an audio tower with an architecture identical to that of a strong pretrained video model. Trained from scratch on hundreds of thousands of hours of raw audio, the audio tower learns to generate realistic sound effects, as well as speech that conveys rich speaker identity and emotion. Fusion is obtained by jointly training the identical video and audio towers via blockwise exchange of timing (via scaled-RoPE embeddings) and semantics (through bidirectional cross-attention) on a vast video corpus. Our model enables cinematic storytelling with natural speech and accurate, context-matched sound effects, producing movie-grade video clips. All the demos, code and model weights are published at https://aaxwaz.github.io/Ovi

Paper Structure

This paper contains 23 sections, 2 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text-To-Video (T2V) Generation
  4. Audio-To-Video (A2V) Generation
  5. Video-To-Audio (V2A) Generation
  6. Joint Audio–Video Generation
  7. Data Processing Pipeline
  8. Data Collection
  9. Audio-Video Data Preprocessing
  10. Pure Audio Data Preprocessing
  11. Method
  12. Architecture Overview
  13. Training Strategy
  14. Audio Model Training
  15. Audio–Video Model Training
  16. ...and 8 more sections

Figures (4)

  • Figure 1: Ovi architecture. Symmetric DiT backbones for audio and video with blockwise, bidirectional cross-attention and shared T5 conditioning from a combined prompt.
  • Figure 2: Cross-modal RoPE affinity matrices before and after scaling. Scaling aligns audio and video temporal positions, improving synchronization.
  • Figure 3: A2V cross-attention visualizations. Heatmaps highlight pixels most attended by audio tokens. Brighter regions correspond to stronger attention.
  • Figure 4: Pairwise win rate (PWR) results of Ovi compared against baselines on Verse-Bench. Higher values indicate stronger human preference for Ovi.