AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis

TL;DR

NVAS seeks to render binaural audio from mono input at arbitrary viewpoints. AV-GS proposes a decoupled architecture that uses 3D Gaussian Splatting (G) to capture geometry and an acoustic field network $\mathcal{F}$ operating on per-point audio-guidance parameters $\alpha$ attached to each Gaussian to generate a pose-specific scene context $\mathcal{C}$, which conditions a binauralizer $\mathcal{B}$ to synthesize left and right channels. It introduces an audio-aware point densification/pruning mechanism and on-the-fly decoding of geometry/material properties to drive binauralization. On RWAVS and SoundSpaces, AV-GS achieves state-of-the-art binaural fidelity and RIR quality with improved efficiency compared to NeRF-based NVAS methods.

Abstract

Novel view acoustic synthesis (NVAS) aims to render binaural audio at any target viewpoint, given a mono audio emitted by a sound source at a 3D scene. Existing methods have proposed NeRF-based implicit models to exploit visual cues as a condition for synthesizing binaural audio. However, in addition to low efficiency originating from heavy NeRF rendering, these methods all have a limited ability of characterizing the entire scene environment such as room geometry, material properties, and the spatial relation between the listener and sound source. To address these issues, we propose a novel Audio-Visual Gaussian Splatting (AV-GS) model. To obtain a material-aware and geometry-aware condition for audio synthesis, we learn an explicit point-based scene representation with an audio-guidance parameter on locally initialized Gaussian points, taking into account the space relation from the listener and sound source. To make the visual scene model audio adaptive, we propose a point densification and pruning strategy to optimally distribute the Gaussian points, with the per-point contribution in sound propagation (e.g., more points needed for texture-less wall surfaces as they affect sound path diversion). Extensive experiments validate the superiority of our AV-GS over existing alternatives on the real-world RWAS and simulation-based SoundSpaces datasets.

Figures (8)

• Figure 1: Sound propagation point patterns between a listener (blue sphere) and emitter (yellow sphere) captured by our AV-GS. Notice the points outside the propagation path (points behind the speaker, points behind rigid walls). Please note we slice the scene into half along the y-axis (omitting the points from the ceiling) in order to facilitate better visibility.
• Figure 2: Overview of our proposed AV-GS. Our model is comprised of a 3D Gaussian Splatting model $G$, an acoustic field network $\mathcal{F}$ and an audio binauralizer $\mathcal{B}$. We first train $G$ to capture the scene geometry information. Next, we construct an audio-focused point representation $G_a$, with the location $X$ and audio-guidance parameter $\alpha$ initialized by the pre-trained $G$. Then the acoustic field network $\mathcal{F}$ is used to process the $\alpha$ parameters for all the Gaussian points in the vicinity of the listener and the sound source (in the 3D space). The output from $\mathcal{F}$ is finally used to condition the audio binauralizer $\mathcal{B}$, which transforms the mono audio to binaural audio w.r.t the listener and sound source location.
• Figure 3: In the presence of (a) complex geometry, and (b) meaningless views, AV-NeRF, when compared to our AV-GS makes errors in binaural synthesis. For both scenarios we showcase the corresponding listener view, used by AV-NeRF, as well as the learned holistic scene representation that is used by AV-GS, and hence unaffected by both scenarios.
• Figure 4: Ablation on the size of vicinity w.r.t the listener and sound source position. Percentile-k denotes the top $k$ % points nearest to the listener and sound source are considered in computing the scene context. (a) RGB color (from $G$), (b) learned $\alpha$ (from $G_a$), (c) 5% percentile vicinity, (d) 25% percentile vicinity.
• Figure 5: Distance aware audio rendering. As the distance from the sound source increases the amplitude of the synthesized binaural audio decreases. Yellow sphere - location of the sound source, blue sphere - location of the listener.