AV-Surf: Surface-Enhanced Geometry-Aware Novel-View Acoustic Synthesis

TL;DR

AV-Surf tackles realistic novel-view acoustic synthesis by fusing rich geometry cues derived from 2D Gaussian Splatting with visual information through a dual cross-attention transformer. A ConvNeXt-based Spectral Refinement Network further enhances the spectral fidelity of binaural outputs. The approach achieves state-of-the-art or competitive results on real-world RWAVS and synthetic SoundSpaces datasets, demonstrated by improvements in MAG, ENV, T60, and C50/EDT across multiple scenes. This geometry-guided, multimodal fusion framework advances spatial acoustics modeling in complex real-world environments and offers practical benefits for AR/VR and immersive audio applications.

Abstract

Accurately modeling sound propagation with complex real-world environments is essential for Novel View Acoustic Synthesis (NVAS). While previous studies have leveraged visual perception to estimate spatial acoustics, the combined use of surface normal and structural details from 3D representations in acoustic modeling has been underexplored. Given their direct impact on sound wave reflections and propagation, surface normals should be jointly modeled with structural details to achieve accurate spatial acoustics. In this paper, we propose a surface-enhanced geometry-aware approach for NVAS to improve spatial acoustic modeling. To achieve this, we exploit geometric priors, such as image, depth map, surface normals, and point clouds obtained using a 3D Gaussian Splatting (3DGS) based framework. We introduce a dual cross-attention-based transformer integrating geometrical constraints into frequency query to understand the surroundings of the emitter. Additionally, we design a ConvNeXt-based spectral features processing network called Spectral Refinement Network (SRN) to synthesize realistic binaural audio. Experimental results on the RWAVS and SoundSpace datasets highlight the necessity of our approach, as it surpasses existing methods in novel view acoustic synthesis.

Figures (7)

• Figure 1: Our proposed AV-Surf learns how acoustics interact with various geometry cues in the real world and uses them to predict sound source leads to binaural audio from a novel-viewpoint.
• Figure 2: AV-Surf overview. AV-Surf first trains 2DGS with given multi-view images to obtain various types of real-world environment information. Then we utilize two types of encoders, $\mathit{E}_{\mathcal{P}}$ and $\mathit{E}_{\mathcal{I}}$, to extract geometry and spatial features, $\mathit{F}_\mathcal{P}$ and $\mathit{F}_\mathcal{I}$. We inject spatial cues to position added frequency embeddings $\mathit{F}_\mathcal{F}$ with iterative transformer layers to learn real world acoustics. After $\mathit{F}_{acoustic}$ is obtained, AV-Surf follows the method from the previous study liang2023av to get mixture $\mathit{M}_m$ and difference $\mathit{M}_d$ acoustic masks. Finally, our ConvNeXt-based Spectral Refinement Network (SRN) takes the sound source and both masks as inputs, refining them to generate realistic novel-view binaural audio.
• Figure 3: We project the spatial-geometric features to frequency dimension and carefully integrated into position aware frequency embeddings by our Transformer architecture. Spatial-geometry aligned $\mathcal{F}_{acoustic}$ feature is then delivered to MLP for generating mixture mask and difference mask.
• Figure 4: SRN estimates waveform based on mixture mask $M_m$, difference mask $M_d$, STFT spectrogram $S_s$, estimated left and right channel spectrogram $S_{l/r}$ by using sequential depth-wise convolution, normalization, point-wise convolution, GELU activation and DropPath.
• Figure 5: Qualitative results on RWAVS dataset. Comparison of AV-Surf, AV-NeRF, and AV-GS against the ground-truth.