Think Hierarchically, Act Dynamically: Hierarchical Multi-modal Fusion and Reasoning for Vision-and-Language Navigation

TL;DR

This paper tackles Vision-and-Language Navigation by addressing the fragmentation of visual, linguistic, and temporal information in prior methods. It proposes MFRA, a Multi-level Fusion and Reasoning Architecture that hierarchically integrates multi-modal features via a DIRformer-based fusion backbone and a dynamic reasoning module with instruction-guided attention. The training objective blends navigation supervision with auxiliary tasks (MLM, MVC, OG) to improve semantic grounding and cross-modal alignment. Empirically, MFRA delivers state-of-the-art performance and robust generalization across multiple VLN benchmarks (REVERIE, SOON, R2R), demonstrating the value of hierarchical fusion and dynamic reasoning for embodied AI.

Abstract

Vision-and-Language Navigation (VLN) aims to enable embodied agents to follow natural language instructions and reach target locations in real-world environments. While prior methods often rely on either global scene representations or object-level features, these approaches are insufficient for capturing the complex interactions across modalities required for accurate navigation. In this paper, we propose a Multi-level Fusion and Reasoning Architecture (MFRA) to enhance the agent's ability to reason over visual observations, language instructions and navigation history. Specifically, MFRA introduces a hierarchical fusion mechanism that aggregates multi-level features-ranging from low-level visual cues to high-level semantic concepts-across multiple modalities. We further design a reasoning module that leverages fused representations to infer navigation actions through instruction-guided attention and dynamic context integration. By selectively capturing and combining relevant visual, linguistic, and temporal signals, MFRA improves decision-making accuracy in complex navigation scenarios. Extensive experiments on benchmark VLN datasets including REVERIE, R2R, and SOON demonstrate that MFRA achieves superior performance compared to state-of-the-art methods, validating the effectiveness of multi-level modal fusion for embodied navigation.

Figures (4)

• Figure 1: Illustration of MFRA selected navigable candidates,which provides crucial information such as attributes and relationships between objects for further VLN reasoning module. Best viewed in color.
• Figure 2: The overall pipeline. (a) The baseline method uses a dual-scale graph transformer to encode the panoramic view, the topological map, and the instruction for action prediction. (b) Our approach incorporates multi-level feature fusion as model input. The representations of each candidate view are obtained with the hierarchical multi-modal fusion, the instruction-guided spatial attention module and the context-aware interaction module. Best viewed in color.
• Figure 3: Visualization of navigation examples. The sentence within the yellow box is the navigation instruction for the agent. We show a comparison where our MFRA chooses the right location while the baseline model makes the wrong choice. Best viewed in color.
• Figure 4: Performance comparison on REVERIE and SOON validation unseen splits. MFRA achieves consistent improvements in Success Rate (SR) and Remote Grounding Success (RGS) over baseline methods.