Enhancing XR Auditory Realism via Multimodal Scene-Aware Acoustic Rendering

TL;DR

Extended Reality audio realism hinges on coherent cross-modal cues between visuals and acoustics. SAMOSA provides an on-device, multimodal pipeline that fuses real-time geometry, material, and acoustic context to synthesize plausible Room Impulse Responses, enabling real-time rendering with low latency and small footprint. The system integrates shoebox-based geometry, material distributions, scene-type embeddings, and RIR synthesis with early reflections and late reverberation, achieving about 58 ms end-to-end latency and a ~3.5 MB model footprint. Objective metrics ($RT_{60}$ and $EDT$) and expert perceptual evaluations (N=12) show SAMOSA outperforms non-adaptive baselines and offers perceptual gains in Naturalness and Externalization without sacrificing Clarity, signaling strong practical impact for on-device XR audio realism.

Abstract

In Extended Reality (XR), rendering sound that accurately simulates real-world acoustics is pivotal in creating lifelike and believable virtual experiences. However, existing XR spatial audio rendering methods often struggle with real-time adaptation to diverse physical scenes, causing a sensory mismatch between visual and auditory cues that disrupts user immersion. To address this, we introduce SAMOSA, a novel on-device system that renders spatially accurate sound by dynamically adapting to its physical environment. SAMOSA leverages a synergistic multimodal scene representation by fusing real-time estimations of room geometry, surface materials, and semantic-driven acoustic context. This rich representation then enables efficient acoustic calibration via scene priors, allowing the system to synthesize a highly realistic Room Impulse Response (RIR). We validate our system through technical evaluation using acoustic metrics for RIR synthesis across various room configurations and sound types, alongside an expert evaluation (N=12). Evaluation results demonstrate SAMOSA's feasibility and efficacy in enhancing XR auditory realism.

Figures (6)

• Figure 1: Overview of the Samosa pipeline. Inputs such as Depth/Plane Feeds and Camera Feeds are processed by three parallel perception modules: Shoebox Estimation, Material Segmentation, and Acoustic Parameter Estimation. The geometry and material outputs are fused in the Room Estimation step. This combined room model, along with the estimated Acoustic Parameters, Headset 6-DoF, and Audio Source 6-DoF, all inform the RIR Synthesis module. The resulting Scene RIR is then convolved with the Input Audio Signal to produce the final Audio Output.
• Figure 2: Visualization of Samosa's dynamic Shoebox Estimation. Figures (a) through (d) illustrate how the estimated shoebox room geometry is updated over time as a user explores a new scene.
• Figure 3: Visualization of Samosa's Material Segmentation. (a) Input binocular egocentric view. (b) Corresponding output segmentation map where pixels are classified into material categories.
• Figure 4: Example application scenarios enabled by Samosa. Applications illustrated are: (a) Rendering a virtual cat in Video-See-Through (VST) mode with plausible environmental acoustics. (b) Enhancing auditory co-presence in XR video conferencing by matching remote user audio to the local space. (c) Creating an immersive video watching experience where ambient sounds blend with the real room. (d) Realistic rendering of audio sources within a detailed VR scene reconstructed via Zip-NeRF zipnerf_2023.
• Figure 5: Expert evaluation results comparing Non-Adaptive, Samosa-GeoOnly, and Samosa (Ours) audio rendering methods across three perceptual metrics (0-10 scale, higher is better). Left: Mean ratings with 95% Confidence Interval (CI) error bars. Right: Median ratings with Interquartile Range (IQR = Q3 - Q1) error bars. Statistical significance of pairwise comparisons is detailed in Figure \ref{['fig:expert_eval_stat_sig']}.