UniVerse-1: Unified Audio-Video Generation via Stitching of Experts

TL;DR

UniVerse-1 addresses the lack of open, synchronized joint audio–video generation by stitching a pre-trained video diffusion model with a music-generation model, enabling bidirectional cross-modal interaction. It introduces an online data-annotation pipeline to ensure precise temporal alignment and identifies and mitigates cross-modal noise correlations through independent noise sampling. The approach is trained on ~7,600 hours of aligned data and evaluated on Verse-Bench, showing competitive performance in synchronous generation and establishing a strong open-source baseline. The work demonstrates a practical, scalable path to high-quality multimodal synthesis by effectively reusing unimodal priors, summarized through extensive ablations and a dedicated benchmark. The release of UniVerse-1 and Verse-Bench aims to accelerate research in open, joint audio–video diffusion.

Abstract

We introduce UniVerse-1, a unified, Veo-3-like model capable of simultaneously generating coordinated audio and video. To enhance training efficiency, we bypass training from scratch and instead employ a stitching of experts (SoE) technique. This approach deeply fuses the corresponding blocks of pre-trained video and music generation experts models, thereby fully leveraging their foundational capabilities. To ensure accurate annotations and temporal alignment for both ambient sounds and speech with video content, we developed an online annotation pipeline that processes the required training data and generates labels during training process. This strategy circumvents the performance degradation often caused by misalignment text-based annotations. Through the synergy of these techniques, our model, after being finetuned on approximately 7,600 hours of audio-video data, produces results with well-coordinated audio-visuals for ambient sounds generation and strong alignment for speech generation. To systematically evaluate our proposed method, we introduce Verse-Bench, a new benchmark dataset. In an effort to advance research in audio-video generation and to close the performance gap with state-of-the-art models such as Veo3, we make our model and code publicly available. We hope this contribution will benefit the broader research community. Project page: https://dorniwang.github.io/UniVerse-1/.

Figures (6)

• Figure 1: Architecture of UniVerse-1. (a) Overall architecture. The architectural foundation of UniVerse-1 is realized through a stitching of experts methodology. This approach deeply integrates the pre-trained Wan2.1 video model and the Ace-step audio model. (b) Fused block. The fusion is implemented at a granular, block-by-block level, where each block in the Wan architecture is deeply fused with its corresponding block in the Ace-step architecture.
• Figure 2: Revised attention of UniVerse-1. (a) Self attention of video branch, with additional mel tokens as input. (b) Cross attention of video branch, with additional mel tokens as input. (c) Cross attention of mel branch, with addition video tokens as input.
• Figure 3: Statistical results of Verse-Bench. Best viewed with zoom-in. A larger, high-resolution version is available in Appendix \ref{['lab:verse-bench-fig']}
• Figure 4: Statistical results of set1 and 2.
• Figure 5: Statistical results of set1.