Residual Cross-Modal Fusion Networks for Audio-Visual Navigation

TL;DR

The paper tackles audio-visual navigation by addressing cross-modal fusion weaknesses that cause modal imbalance and poor cross-domain generalization. It introduces the Cross-Modal Residual Fusion Network (CRFN), which enforces bidirectional residual interactions between visual and audio streams and couples them with a lightweight Fusion Controller that uses learnable residual scaling factors $\beta_v$ and $\beta_a$ to adaptively regulate modality contributions. The interaction vector $h_{interact}$, computed as $\frac{1}{2}(U_v(v_t) + U_a(a_t))$, enables fine-grained reciprocal refinement while preserving modality independence, with normalization and tanh activation for stability. Empirical results on Replica and Matterport3D show CRFN outperforms baselines and generalizes across domains, and analysis reveals a dynamic modality-dependence phenomenon: visual dominance in synthetic environments and cross-modal complementarity in real-world scenes, offering new insights into multimodal collaboration in embodied agents.

Abstract

Audio-visual embodied navigation aims to enable an agent to autonomously localize and reach a sound source in unseen 3D environments by leveraging auditory cues. The key challenge of this task lies in effectively modeling the interaction between heterogeneous features during multimodal fusion, so as to avoid single-modality dominance or information degradation, particularly in cross-domain scenarios. To address this, we propose a Cross-Modal Residual Fusion Network, which introduces bidirectional residual interactions between audio and visual streams to achieve complementary modeling and fine-grained alignment, while maintaining the independence of their representations. Unlike conventional methods that rely on simple concatenation or attention gating, CRFN explicitly models cross-modal interactions via residual connections and incorporates stabilization techniques to improve convergence and robustness. Experiments on the Replica and Matterport3D datasets demonstrate that CRFN significantly outperforms state-of-the-art fusion baselines and achieves stronger cross-domain generalization. Notably, our experiments also reveal that agents exhibit differentiated modality dependence across different datasets. The discovery of this phenomenon provides a new perspective for understanding the cross-modal collaboration mechanism of embodied agents.

Figures (6)

• Figure 1: Humans can effortlessly integrate visual and auditory information, yet agents often suffer from performance degradation due to modal imbalance or domain discrepancy.
• Figure 2: Illustration of the basic architecture. The agent's navigation process operates in three stages: (1) Observation and Encoder, where the agent processes visual (RGB or Depth) and auditory (spectrogram) inputs via respective encoders to extract features; (2) Interaction and Integration, where our proposed CRFN module performs bidirectional residual updates to refine features and adaptively balances them using a Fusion Controller; and (3) Policy Network, where the fused representation is fed into a GRU-based Actor-Critic model to capture temporal dependencies and predict the final navigation action $a_t$.
• Figure 3: Architecture of the Cross-Modal Feature Fusion Module. This module enables visual and audio features to refine and complement each other through bidirectional residual paths. Features of each modality are updated and mutually influenced in the bidirectional interaction, ensuring a balanced exchange of information.
• Figure 4: Navigation trajectories on the top-down map in the Replica scenes. Agent paths transition from dark to light blue temporally, while green indicates the shortest geodesic path.
• Figure 5: Navigation trajectories on the top-down map in the Matterport3D scenes. Agent paths transition from dark to light blue temporally, while green indicates the shortest geodesic path.