Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Omni2Sound: Towards Unified Video-Text-to-Audio Generation

Yusheng Dai, Zehua Chen, Yuxuan Jiang, Baolong Gao, Qiuhong Ke, Jun Zhu, Jianfei Cai

TL;DR

A three-stage multi-task progressive training schedule is designed that converts cross-task competition into joint optimization and mitigates modality bias in the VT2A task, maintaining both audio-visual alignment and off-screen audio generation faithfulness.

Abstract

Training a unified model integrating video-to-audio (V2A), text-to-audio (T2A), and joint video-text-to-audio (VT2A) generation offers significant application flexibility, yet faces two unexplored foundational challenges: (1) the scarcity of high-quality audio captions with tight A-V-T alignment, leading to severe semantic conflict between multimodal conditions, and (2) cross-task and intra-task competition, manifesting as an adverse V2A-T2A performance trade-off and modality bias in the VT2A task. First, to address data scarcity, we introduce SoundAtlas, a large-scale dataset (470k pairs) that significantly outperforms existing benchmarks and even human experts in quality. Powered by a novel agentic pipeline, it integrates Vision-to-Language Compression to mitigate visual bias of MLLMs, a Junior-Senior Agent Handoff for a 5 times cost reduction, and rigorous Post-hoc Filtering to ensure fidelity. Consequently, SoundAtlas delivers semantically rich and temporally detailed captions with tight V-A-T alignment. Second, we propose Omni2Sound, a unified VT2A diffusion model supporting flexible input modalities. To resolve the inherent cross-task and intra-task competition, we design a three-stage multi-task progressive training schedule that converts cross-task competition into joint optimization and mitigates modality bias in the VT2A task, maintaining both audio-visual alignment and off-screen audio generation faithfulness. Finally, we construct VGGSound-Omni, a comprehensive benchmark for unified evaluation, including challenging off-screen tracks. With a standard DiT backbone, Omni2Sound achieves unified SOTA performance across all three tasks within a single model, demonstrating strong generalization across benchmarks with heterogeneous input conditions. The project page is at https://swapforward.github.io/Omni2Sound.

Omni2Sound: Towards Unified Video-Text-to-Audio Generation

TL;DR

A three-stage multi-task progressive training schedule is designed that converts cross-task competition into joint optimization and mitigates modality bias in the VT2A task, maintaining both audio-visual alignment and off-screen audio generation faithfulness.

Abstract

Training a unified model integrating video-to-audio (V2A), text-to-audio (T2A), and joint video-text-to-audio (VT2A) generation offers significant application flexibility, yet faces two unexplored foundational challenges: (1) the scarcity of high-quality audio captions with tight A-V-T alignment, leading to severe semantic conflict between multimodal conditions, and (2) cross-task and intra-task competition, manifesting as an adverse V2A-T2A performance trade-off and modality bias in the VT2A task. First, to address data scarcity, we introduce SoundAtlas, a large-scale dataset (470k pairs) that significantly outperforms existing benchmarks and even human experts in quality. Powered by a novel agentic pipeline, it integrates Vision-to-Language Compression to mitigate visual bias of MLLMs, a Junior-Senior Agent Handoff for a 5 times cost reduction, and rigorous Post-hoc Filtering to ensure fidelity. Consequently, SoundAtlas delivers semantically rich and temporally detailed captions with tight V-A-T alignment. Second, we propose Omni2Sound, a unified VT2A diffusion model supporting flexible input modalities. To resolve the inherent cross-task and intra-task competition, we design a three-stage multi-task progressive training schedule that converts cross-task competition into joint optimization and mitigates modality bias in the VT2A task, maintaining both audio-visual alignment and off-screen audio generation faithfulness. Finally, we construct VGGSound-Omni, a comprehensive benchmark for unified evaluation, including challenging off-screen tracks. With a standard DiT backbone, Omni2Sound achieves unified SOTA performance across all three tasks within a single model, demonstrating strong generalization across benchmarks with heterogeneous input conditions. The project page is at https://swapforward.github.io/Omni2Sound.
Paper Structure (46 sections, 2 equations, 7 figures, 9 tables)

This paper contains 46 sections, 2 equations, 7 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Audio Caption Dataset.
  4. Unified Audio Generation Model.
  5. SoundAtlas: V-A-T Data Construction
  6. A-V Consistency Routing.
  7. Vision-to-Language Compression.
  8. Junior–Senior Agent Handoff.
  9. Post-hoc Filtering and Verification.
  10. Comparison with Existing Pipeline
  11. Omni2Sound: Unified VT2A Generation
  12. Foundation Model Architecture
  13. Three-stage Progressive Multi-task Training
  14. Stage 1: Large-scale T2A Pretraining.
  15. Stage 2: Multi-task Interleaved Training.
  16. ...and 31 more sections

Figures (7)

  • Figure 1: Challenges in scaling high-quality audio captions.
  • Figure 2: Data Construction Pipeline of SoundAtlas (Left). Comparison against SOTA baselines and human annotations (Right) .
  • Figure 3: Overview of our unified VT2A framework, which integrates global semantics and temporal alignment, supporting flexible T2A, V2A, and VT2A generation.
  • Figure 4: Subjective Evaluation Results on VGGSound-Omni. We report Mean Opinion Scores (MOS) on a 1-5 scale across three dimensions: Acoustic Quality (MOS-Q), Semantic Alignment (MOS-S), and Temporal Alignment (MOS-T). Omni2Sound consistently outperforms competitive baselines (AudioX, MMAudio, HunyuanVideo-Foley, Frienren-V2A) across all perceptual metrics on both VT2A and VT2A tasks, validating its superior generation fidelity and alignment.
  • Figure 5: Audio Captioning Instruction for SoundAtlas.
  • ...and 2 more figures