UNeMo: Collaborative Visual-Language Reasoning and Navigation via a Multimodal World Model

TL;DR

UNeMo tackles VLN by tightly coupling visual-state reasoning with navigation policy through a Multimodal World Model (MWM) and a Hierarchical Prediction-Feedback Navigator (HPFN). The MWM predicts future visual states from current visual, linguistic, and action cues, while HPFN updates node embeddings and refines action decisions in a closed loop, enabling dynamic bidirectional improvement. Empirical results on R2R and REVERIE show state-of-the-art or competitive performance, with notable gains in unseen environments and robustness for long-horizon navigation, while maintaining efficiency with compact backbones. The work demonstrates the practical value of joint multimodal state reasoning for robust, generalizable VLN and points to future extensions in real-world robotics and more complex instructions.

Abstract

Vision-and-Language Navigation (VLN) requires agents to autonomously navigate complex environments via visual images and natural language instruction--remains highly challenging. Recent research on enhancing language-guided navigation reasoning using pre-trained large language models (LLMs) has shown promising prospects. However, the reasoning of such methods is limited to the linguistic modality, lacking visual reasoning capabilities. Moreover, existing reasoning modules are optimized separately from navigation policies, leading to incompatibility and potential conflicts in optimization objectives. To tackle these challenges, we introduce UNeMo, a novel framework designed for the collaborative optimization of visual state reasoning and navigational decision-making. It introduces a Multimodal World Model (MWM) that takes visual features, language instructions, and navigational actions as inputs to jointly predict subsequent visual states, enabling cross-modal reasoning. Via a Hierarchical Prediction-Feedback (HPN) mechanism, MWM collaborates with navigation policies: the first layer generates actions using current vision-and-language features; MWM then infers post-action visual states to guide the second layer's fine-grained decisions. This forms a dynamic bidirectional promotion mechanism where MWM reasoning optimizes navigation policies, while policy decisions feedback to improve MWM's reasoning accuracy. Experiments on R2R and REVERIE datasets show UNeMo outperforms state-of-the-art methods by 2.1% and 0.7% in navigation accuracy for unseen scenes, validating its effectiveness.

Figures (4)

• Figure 1: Comparison of main differences between UNeMo and NavGPT2. UNeMo introduces MWM for visual state reasoning and achieves joint optimization with the navigation policy via a hierarchical predictive-feedback navigator.
• Figure 2: Overview of the UNeMo framework. Similar to NavGPT2, the input visual observation $O_t$ and instruction Tare processed through a pre-trained LLM encoder to obtain visual and word features, respectively. UNeMo introduces two key components: (1) The Multimodal State Reasoning module receives partial-view representations of the highest-scoring action node and predicts its complete post-execution visual state; (2) The Decision Making module fuses these state-reasoning results with linguistic features for final navigation action selection.
• Figure 3: Qualitative comparison between MWM predicted features and ground-truth labels under partial observations
• Figure 4: Quantitative evaluation of MWM predictions: Distributions of cosine similarity and MSE against labels in val-unseen