DynFOA: Generating First-Order Ambisonics with Conditional Diffusion for Dynamic and Acoustically Complex 360-Degree Videos

TL;DR

DynFOA addresses the challenge of generating high-fidelity, physically grounded FOA from 360-degree video in dynamic, acoustically complex scenes. It fuses geometry- and material-aware visual scene reconstruction with a conditional diffusion generator that accounts for occlusion, reflections, and reverberation, guided by a three-core architecture: a Video Encoder, an Audio Encoder, and a Conditional Diffusion Generator. Key contributions include the 3D Gaussian Splatting-based scene reconstruction with per-surface acoustic properties, a latent-domain diffusion model conditioned on geometry/material and propagation cues, and the Dyn360 dataset for comprehensive evaluation. The approach yields superior spatial accuracy, acoustic fidelity, and distribution matching, and supports real-time, head-tracked FOA rendering, enhancing immersion in VR and immersive media applications.

Abstract

Spatial audio is crucial for creating compelling immersive 360-degree video experiences. However, generating realistic spatial audio, such as first-order ambisonics (FOA), from 360-degree videos in complex acoustic scenes remains challenging. Existing methods often overlook the dynamic nature and acoustic complexity of 360-degree scenes, fail to fully account for dynamic sound sources, and neglect complex environmental effects such as occlusion, reflections, and reverberation, which are influenced by scene geometries and materials. We propose DynFOA, a framework based on dynamic acoustic perception and conditional diffusion, for generating high-fidelity FOA from 360-degree videos. DynFOA first performs visual processing via a video encoder, which detects and localizes multiple dynamic sound sources, estimates their depth and semantics, and reconstructs the scene geometry and materials using a 3D Gaussian Splatting. This reconstruction technique accurately models occlusion, reflections, and reverberation based on the geometries and materials of the reconstructed 3D scene and the listener's viewpoint. The audio encoder then captures the spatial motion and temporal 4D sound source trajectories to fine-tune the diffusion-based FOA generator. The fine-tuned FOA generator adjusts spatial cues in real time, ensuring consistent directional fidelity during listener head rotation and complex environmental changes. Extensive evaluations demonstrate that DynFOA consistently outperforms existing methods across metrics such as spatial accuracy, acoustic fidelity, and distribution matching, while also improving the user experience. Therefore, DynFOA provides a robust and scalable approach to rendering realistic dynamic spatial audio for VR and immersive media applications.

Figures (3)

• Figure 1: Overview of Our DynFOA. The model features a three-core architecture: audio encoder, video encoder, and conditional diffusion generator. The audio encoder enhances FOA audio robustness against occlusion, reflections, and reverberation through dynamic sound source processing. The video encoder reconstructs video information to support immersive FOA experiences. The conditional diffusion integrates scene context for deep FOA encoding and multimodal fusion, enabling real-time, high-fidelity immersive FOA rendering in scenes with occlusion, reflections, and reverberation.
• Figure 2: The piano performance performed by a singer and pianist in an indoor setting. Due to the occlusion and distribution of objects, the environment exhibits significant occlusion, reflections, and reverberation characteristics. Existing methods often reduce the clarity and immersiveness of FOA. However, our DynFOA effectively accounts for factors affecting audio quality in this complex scene, accurately capturing spatial cues while preserving the natural timbre of instruments and vocals. Compared to existing baseline methods, our DynFOA achieves more realistic sound reproduction, delivering a more immersive experience.
• Figure 3: The dance hall events involve multiple simultaneous sound sources and complex spatial dynamics. This environment presents numerous challenges, such as overlapping vocalizations, dynamic motion of sound sources, and highly variable reverberation patterns. Our DynFOA effectively separates these concurrent audio elements, reconstructing the scene's spatial distribution with superior accuracy while maintaining spectral consistency. This demonstrates our DynFOA exceptional robustness in handling highly complex real-world acoustic conditions, outperforming baseline models.