VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation

TL;DR

VinTAGe tackles the challenge of holistic audio generation by conditioning on both text and video to produce onscreen and offscreen sounds that are temporally synchronized and semantically aligned. It introduces a flow-based transformer architecture with a Visual-Text Encoder and a Joint VT-SiT generator, augmented by teacher-guided alignment to mitigate modality bias. The authors also present VinTAGe-Bench, a new dataset of 636 video-text-audio pairs containing onscreen and offscreen sounds, enabling robust evaluation beyond traditional V2A or T2A tasks. Across experiments, VinTAGe achieves state-of-the-art results on VGGSound and demonstrates the value of joint video-text conditioning for holistic audio synthesis, with extensive ablations validating each component's contribution.

Abstract

Recent advances in audio generation have focused on text-to-audio (T2A) and video-to-audio (V2A) tasks. However, T2A or V2A methods cannot generate holistic sounds (onscreen and off-screen). This is because T2A cannot generate sounds aligning with onscreen objects, while V2A cannot generate semantically complete (offscreen sounds missing). In this work, we address the task of holistic audio generation: given a video and a text prompt, we aim to generate both onscreen and offscreen sounds that are temporally synchronized with the video and semantically aligned with text and video. Previous approaches for joint text and video-to-audio generation often suffer from modality bias, favoring one modality over the other. To overcome this limitation, we introduce VinTAGe, a flow-based transformer model that jointly considers text and video to guide audio generation. Our framework comprises two key components: a Visual-Text Encoder and a Joint VT-SiT model. To reduce modality bias and improve generation quality, we employ pretrained uni-modal text-to-audio and video-to-audio generation models for additional guidance. Due to the lack of appropriate benchmarks, we also introduce VinTAGe-Bench, a dataset of 636 video-text-audio pairs containing both onscreen and offscreen sounds. Our comprehensive experiments on VinTAGe-Bench demonstrate that joint text and visual interaction is necessary for holistic audio generation. Furthermore, VinTAGe achieves state-of-the-art results on the VGGSound benchmark. Our source code and pre-trained models will be released. Demo is available at: https://www.youtube.com/watch?v=QmqWhUjPkJI.

Figures (18)

• Figure 1: Our VinTAGe model can generate visually aligned and text-corresponding sounds, including both onscreen and offscreen sound sources, providing a more holistic audio experience. However, V2A models can only generate video-aligned but semantically incomplete sounds, while T2A models, though generating semantically complete sounds, are not visually aligned.
• Figure 2: (A) VT-Encoder, (B) Joint VT-SiT block, (C) Overall training pipeline.
• Figure 3: Input and target construction. During training, an audio latent $x$ is interpolated at time t (sampled from uniform distribution), with random noise sampled from gaussion $\epsilon$ and generates noisy input and target velocity.
• Figure 4: Inference pipeline.
• Figure 5: The statistics and examples of our VinTAGe-Bench.