SpatialV2A: Visual-Guided High-fidelity Spatial Audio Generation

TL;DR

This work addresses the lack of spatial realism in video-to-audio generation by introducing BinauralVGGSound, a large-scale binaural video dataset, and SpatialV2A, a dual-branch framework that learns visual-guided spatial audio via Conditional Flow Matching. SpatialV2A decouples left and right channel synthesis while integrating a visual-guided spatialization module that extracts spatial cues from video to condition the generative process. Experiments show SpatialV2A achieves superior spatial fidelity and maintains semantic and temporal coherence, outperforming state-of-the-art mono and spatial baselines in both objective metrics and subjective assessments. The contributions enable immersive, perceptually realistic audio-visual experiences and provide a scalable dataset and methodology for future spatial V2A research.

Abstract

While video-to-audio generation has achieved remarkable progress in semantic and temporal alignment, most existing studies focus solely on these aspects, paying limited attention to the spatial perception and immersive quality of the synthesized audio. This limitation stems largely from current models' reliance on mono audio datasets, which lack the binaural spatial information needed to learn visual-to-spatial audio mappings. To address this gap, we introduce two key contributions: we construct BinauralVGGSound, the first large-scale video-binaural audio dataset designed to support spatially aware video-to-audio generation; and we propose a end-to-end spatial audio generation framework guided by visual cues, which explicitly models spatial features. Our framework incorporates a visual-guided audio spatialization module that ensures the generated audio exhibits realistic spatial attributes and layered spatial depth while maintaining semantic and temporal alignment. Experiments show that our approach substantially outperforms state-of-the-art models in spatial fidelity and delivers a more immersive auditory experience, without sacrificing temporal or semantic consistency. All datasets, code, and model checkpoints will be publicly released to facilitate future research.

Figures (5)

• Figure 1: Comparison of traditional V2A models and the proposed SpatialV2A. (a) Traditional V2A generates mono audio. (b) SpatialV2A produces spatially consistent binaural audio.
• Figure 2: Overall pipeline of BinauralVGGSound dataset construction.
• Figure 3: Example visualization from BinauralVGGSound for a flute-playing scene. The transition of bounding boxes from green to blue reflects the estimated sound source movement, demonstrating spatial consistency with the visual motion.
• Figure 4: Overall architecture of SpatialV2A. The framework employs a dual-branch design for spatial audio generation (Sec. \ref{['Dual_gen_frame']}) and integrates a visual-guided audio spatialization module (Sec. \ref{['spatial_module']}) that injects video-derived spatial cues to enhance spatial perception.
• Figure 5: Mel-spectrogram comparison between our method, baselines, original mono GT, and our binaural GT. Our approach better preserves salient audio effects while reducing background noise.