Iterative Residual Cross-Attention Mechanism: An Integrated Approach for Audio-Visual Navigation Tasks

TL;DR

The paper tackles audiovisual navigation by replacing traditional modular fusion and GRU-based sequence modeling with an end-to-end Iterative Residual Cross-Attention Module (IRCAM). The proposed IRCAM-AVN unifies feature fusion and temporal modeling via multi-level residual cross-attention, improving bias robustness and stability. Empirical evaluations on Replica and Matterport3D show state-of-the-art improvements across SR, SPL, and SNA, with ablations confirming the importance of the RT, PE, and EN components, and visualizations illustrating deeper cross-modal integration. This approach advances robust, efficient navigation in complex 3D environments and has practical implications for real-world audio-visual embodied agents.

Abstract

Audio-visual navigation represents a significant area of research in which intelligent agents utilize egocentric visual and auditory perceptions to identify audio targets. Conventional navigation methodologies typically adopt a staged modular design, which involves first executing feature fusion, then utilizing Gated Recurrent Unit (GRU) modules for sequence modeling, and finally making decisions through reinforcement learning. While this modular approach has demonstrated effectiveness, it may also lead to redundant information processing and inconsistencies in information transmission between the various modules during the feature fusion and GRU sequence modeling phases. This paper presents IRCAM-AVN (Iterative Residual Cross-Attention Mechanism for Audiovisual Navigation), an end-to-end framework that integrates multimodal information fusion and sequence modeling within a unified IRCAM module, thereby replacing the traditional separate components for fusion and GRU. This innovative mechanism employs a multi-level residual design that concatenates initial multimodal sequences with processed information sequences. This methodological shift progressively optimizes the feature extraction process while reducing model bias and enhancing the model's stability and generalization capabilities. Empirical results indicate that intelligent agents employing the iterative residual cross-attention mechanism exhibit superior navigation performance.

Figures (6)

• Figure 1: A sketched comparison between the previous mainstream method (a) and the proposed IRCAM (b).
• Figure 2: Audio-Visual Navigation Network. Our model leverages acoustic and visual cues from the 3D environment to achieve efficient navigation through a novel iterative residual cross-attention mechanism.
• Figure 3: Our audiovisual embodied navigation architecture is compared with previous works on embodied navigation in 3D environments. Earlier audiovisual navigation methods typically perform feature fusion and GRU operations in separate stages. However, this staged processing often results in uncoordinated information flow and insufficient feature integration, ultimately leading to inefficient pathfinding.
• Figure 4: An intuitive comparison between our model and various baselines on the audio-visual navigation task in the Replica and Matterport3D datasets.
• Figure 5: Our audiovisual embodied navigation architecture is also compared with a version of the architecture where the EN module is ablated. It is observed that without the EN module, the attention mechanism struggles to distinguish and focus on critical regions effectively.