SoundVista: Novel-View Ambient Sound Synthesis via Visual-Acoustic Binding

TL;DR

SoundVista tackles novel-view ambient-sound synthesis by learning a transfer from sparse reference recordings and panoramic visuals to a target viewpoint. It introduces a Visual-Acoustic Binding module to map visuals to local acoustic properties, a Reference Location Sampler to choose representative reference points, a Reference Integration Transformer to adaptively weight references, and a Spatial Audio Renderer to produce the final binaural audio. Validated on real-world N2S data and Matterport3D Soundspaces, SoundVista achieves state-of-the-art performance across multiple metrics and demonstrates strong generalization to unseen environments. The approach enables scalable, layout-agnostic ambient audio rendering for immersive experiences without requiring granular source-level information.

Abstract

We introduce SoundVista, a method to generate the ambient sound of an arbitrary scene at novel viewpoints. Given a pre-acquired recording of the scene from sparsely distributed microphones, SoundVista can synthesize the sound of that scene from an unseen target viewpoint. The method learns the underlying acoustic transfer function that relates the signals acquired at the distributed microphones to the signal at the target viewpoint, using a limited number of known recordings. Unlike existing works, our method does not require constraints or prior knowledge of sound source details. Moreover, our method efficiently adapts to diverse room layouts, reference microphone configurations and unseen environments. To enable this, we introduce a visual-acoustic binding module that learns visual embeddings linked with local acoustic properties from panoramic RGB and depth data. We first leverage these embeddings to optimize the placement of reference microphones in any given scene. During synthesis, we leverage multiple embeddings extracted from reference locations to get adaptive weights for their contribution, conditioned on target viewpoint. We benchmark the task on both publicly available data and real-world settings. We demonstrate significant improvements over existing methods.

Figures (9)

• Figure 1: SoundVista: a novel method that synthesizes binaural ambient sound for arbitrary scenes from novel viewpoints. Our method leverages pre-acquired audio recordings and visual data captured from sparsely distributed reference points and synthesizes binaural audio consistent with the target 3D position and pose.
• Figure 2: Details of the SoundVista Pipeline: (a) The reference location sampler selects optimal reference locations leveraging embeddings from visual-acoustic binding (VAB). (b) The reference integration transformer uses VAB embeddings to derive contribution weights for each reference. (c) Reweighting by contribution weights adjusts and integrates reference recording channels and pose conditioning for precise sound synthesis. (d) The spatial audio renderer converts reweighted channels and conditions to binaural sound at the target viewpoint.
• Figure 3: Comparison of Qualitative Results: First row: Loudness Map (black triangle: sources; purple triangle: target viewpoint). Second row: Reconstructed binaural waveform at target viewpoint (red: prediction; green: GT). Third row: LRE (lower for better binaural effect).
• Figure 4: STFT Error Curve for different reference densities (left) and training location data ratios (right).
• Figure 5: Visualization Analysis. First three columns: Clustering results for loc only, Vis - w/o VAB (ours without VAB pretraining), and Vis - w/ VAB (ours). Colors indicate cluster regions. VAB helps clustering to align better with room partitions. Last two columns: Attention weights visualization for references, with colored stars (size proportional to weights) and a purple triangle for the target.