A Dual Semantic-Aware Recurrent Global-Adaptive Network For Vision-and-Language Navigation

TL;DR

This work tackles Vision-and-Language Navigation by explicitly mining guiding semantics from both visual and linguistic inputs and by maintaining reasoning continuity across steps. The proposed Dual Semantic-Aware Recurrent Global-Adaptive Network (DSRG) couples an Instruction-Guidance Linguistic Module (IGL) with an Appearance-Semantics Visual Module (ASV), and augments memory via Global Adaptive Aggregation (GAA) and Recurrent Memory Fusion (RMF). Through these components, DS RG achieves state-of-the-art results on R2R and REVERIE, demonstrating improved semantic understanding, robust cross-modal fusion, and effective long-horizon reasoning. The approach offers strong potential for real-world VLN and related embodied AI tasks, with extensive ablations confirming the contribution of each module and memory mechanism.

Abstract

Vision-and-Language Navigation (VLN) is a realistic but challenging task that requires an agent to locate the target region using verbal and visual cues. While significant advancements have been achieved recently, there are still two broad limitations: (1) The explicit information mining for significant guiding semantics concealed in both vision and language is still under-explored; (2) The previously structured map method provides the average historical appearance of visited nodes, while it ignores distinctive contributions of various images and potent information retention in the reasoning process. This work proposes a dual semantic-aware recurrent global-adaptive network (DSRG) to address the above problems. First, DSRG proposes an instruction-guidance linguistic module (IGL) and an appearance-semantics visual module (ASV) for boosting vision and language semantic learning respectively. For the memory mechanism, a global adaptive aggregation module (GAA) is devised for explicit panoramic observation fusion, and a recurrent memory fusion module (RMF) is introduced to supply implicit temporal hidden states. Extensive experimental results on the R2R and REVERIE datasets demonstrate that our method achieves better performance than existing methods. Code is available at https://github.com/CrystalSixone/DSRG.

Figures (7)

• Figure 1: Illustration of the proposed DSRG, which first augments semantics of visual and linguistic inputs respectively, and employs the cross-modal reasoning network along with the recurrent state fusion module and the memory graph to predict the action.
• Figure 2: Overview of the proposed DSRG structure, which includes three components: (a) semantic-aware visual environment learning, (b) guidance-aware linguistic instruction learning, and (c) recurrent global-local visual-linguistic feature fusion.
• Figure 3: Word clouds of (a) landmark and (b) direction tokens.
• Figure 4: Illustration of the instruction-guidance linguistic module.
• Figure 5: Illustration of the appearance-semantics visual module.