Reliability-Aware Geometric Fusion for Robust Audio-Visual Navigation

Abstract

Audio-Visual Navigation (AVN) requires an embodied agent to navigate toward a sound source by utilizing both vision and binaural audio. A core challenge arises in complex acoustic environments, where binaural cues become intermittently unreliable, particularly when generalizing to previously unheard sound categories. To address this, we propose RAVN (Reliability-Aware Audio-Visual Navigation), a framework that conditions cross-modal fusion on audio-derived reliability cues, dynamically calibrating the integration of audio and visual inputs. RAVN introduces an Acoustic Geometry Reasoner (AGR) that is trained with geometric proxy supervision. Using a heteroscedastic Gaussian NLL objective, AGR learns observation-dependent dispersion as a practical reliability cue, eliminating the need for geometric labels during inference. Additionally, we introduce Reliability-Aware Geometric Modulation (RAGM), which converts the learned cue into a soft gate to modulate visual features, thereby mitigating cross-modal conflicts. We evaluate RAVN on SoundSpaces using both Replica and Matterport3D environments, and the results show consistent improvements in navigation performance, with notable robustness in the challenging unheard sound setting.

Figures (4)

• Figure 1: Paradigm comparison. (a) naive deterministic fusion vs. (b) our reliability-aware RAVN framework.
• Figure 2: The RAVN Framework. Multi-modal observations $o_t$ are encoded into $f_a$ and $f_v$. The AGR module estimates geometric embeddings $g_t$ and predictive uncertainty ($\mu_t, \sigma_t$), supervised by Ground Truth (GT) labels via an auxiliary loss. The RAGM module then uses $g_t$ to dynamically modulate visual features into $f'_v$. A recurrent policy (GRU) aggregates these components into the hidden state $s_t$ for end-to-end action ($a_t$) selection.
• Figure 3: RAGM module architecture. Geometric features are transformed into a reliability mask, which applies element-wise modulation to visual features to produce modified visual representations.
• Figure 4: Qualitative results. (a) Top-down trajectory comparisons between the AV-NaV baseline and our RAVN in representative Replica and Mp3D episodes, with SPL shown for each run. (b) Additional rollouts of RAVN across various layouts and start-goal configurations.