Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Physics-Aware Novel-View Acoustic Synthesis with Vision-Language Priors and 3D Acoustic Environment Modeling

Congyi Fan, Jian Guan, Youtian Lin, Dongli Xu, Tong Ye, Qiaoxi Zhu, Pengming Feng, Wenwu Wang

TL;DR

This work tackles the challenge of novel-view acoustic synthesis by proposing Phys-NVAS, a physics-aware framework that jointly models global 3D room geometry and semantic priors about objects and materials. It combines 3D Gaussian Splatting based geometry with a vision-language model to capture absorption and reflection properties, fusing these cues into a physics-aware representation for binaural audio generation. Evaluated on the RWAVS dataset, Phys-NVAS achieves lower MAG and ENV errors than state-of-the-art baselines, demonstrating improved spatial realism and physical consistency. The approach enhances immersive audio by leveraging both geometric and semantic scene information for accurate sound propagation and reverberation modeling.

Abstract

Spatial audio is essential for immersive experiences, yet novel-view acoustic synthesis (NVAS) remains challenging due to complex physical phenomena such as reflection, diffraction, and material absorption. Existing methods based on single-view or panoramic inputs improve spatial fidelity but fail to capture global geometry and semantic cues such as object layout and material properties. To address this, we propose Phys-NVAS, the first physics-aware NVAS framework that integrates spatial geometry modeling with vision-language semantic priors. A global 3D acoustic environment is reconstructed from multi-view images and depth maps to estimate room size and shape, enhancing spatial awareness of sound propagation. Meanwhile, a vision-language model extracts physics-aware priors of objects, layouts, and materials, capturing absorption and reflection beyond geometry. An acoustic feature fusion adapter unifies these cues into a physics-aware representation for binaural generation. Experiments on RWAVS demonstrate that Phys-NVAS yields binaural audio with improved realism and physical consistency.

Physics-Aware Novel-View Acoustic Synthesis with Vision-Language Priors and 3D Acoustic Environment Modeling

TL;DR

This work tackles the challenge of novel-view acoustic synthesis by proposing Phys-NVAS, a physics-aware framework that jointly models global 3D room geometry and semantic priors about objects and materials. It combines 3D Gaussian Splatting based geometry with a vision-language model to capture absorption and reflection properties, fusing these cues into a physics-aware representation for binaural audio generation. Evaluated on the RWAVS dataset, Phys-NVAS achieves lower MAG and ENV errors than state-of-the-art baselines, demonstrating improved spatial realism and physical consistency. The approach enhances immersive audio by leveraging both geometric and semantic scene information for accurate sound propagation and reverberation modeling.

Abstract

Spatial audio is essential for immersive experiences, yet novel-view acoustic synthesis (NVAS) remains challenging due to complex physical phenomena such as reflection, diffraction, and material absorption. Existing methods based on single-view or panoramic inputs improve spatial fidelity but fail to capture global geometry and semantic cues such as object layout and material properties. To address this, we propose Phys-NVAS, the first physics-aware NVAS framework that integrates spatial geometry modeling with vision-language semantic priors. A global 3D acoustic environment is reconstructed from multi-view images and depth maps to estimate room size and shape, enhancing spatial awareness of sound propagation. Meanwhile, a vision-language model extracts physics-aware priors of objects, layouts, and materials, capturing absorption and reflection beyond geometry. An acoustic feature fusion adapter unifies these cues into a physics-aware representation for binaural generation. Experiments on RWAVS demonstrate that Phys-NVAS yields binaural audio with improved realism and physical consistency.
Paper Structure (12 sections, 6 equations, 3 figures, 2 tables)

This paper contains 12 sections, 6 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. 3D Acoustic Environment Modeling and Multi-View Spatial Perception
  4. Physics-Aware Vision-Language Priors
  5. Binaural Audio Synthesis with Physics-Aware Feature Representation
  6. Experiments and Results
  7. Experiment Setup
  8. Performance Comparison
  9. Visualization Analysis
  10. Ablation Study
  11. Conclusion
  12. Acknowledgement

Figures (3)

  • Figure 1: Illustration of scene semantics influencing acoustics. Different materials (e.g., wooden, flat-screen) affect absorption, while objects and their layouts (e.g., In front of the TV, there is a wooden coffee table) modify reflection paths. These semantic cues are often ignored in existing NVAS methods, leading to physically inconsistent audio.
  • Figure 2: Overview of the proposed physics-aware NVAS framework. 3D acoustic environment modeling with 3DGS and depth estimation on multi-view images, enhancing spatial awareness by recovering room geometry and size. Physics-aware vision–language priors further enrich acoustic modeling with object, layout, and material cues that capture absorption and reflection effects. Finally, geometric and semantic features are fused into a unified physics-aware feature representation, enabling realistic and physically consistent binaural audio generation.
  • Figure 3: Comparison of reconstructed binaural waveforms at a target listener position using AV-NeRF and our Phys-NVAS.