Towards Open Environments and Instructions: General Vision-Language Navigation via Fast-Slow Interactive Reasoning

TL;DR

This work tackles open-world generalization in Vision-Language Navigation (VLN) by introducing General Scene Adaptation for VLN (GSA-VLN) and the slow4fast-VLN framework, which tightly couples fast, real-time decision-making with slow, reflective reasoning. The fast module (π) outputs actions based on real-time input and builds a memory history, while the slow module analyzes memories to extract generalized experiences that continuously empower the fast module; instruction-style conversion via Chain-of-Thought prompts enables handling diverse user-language styles. An Experience Library stores distilled experiences as structured entities, enabling retrieval and fusion via attention to guide navigation in unseen environments. Extensive experiments on the GSA-R2R dataset demonstrate robust improvements in in-distribution and out-of-distribution settings, with ablations confirming the contribution of fast-slow interaction and instruction-style conversion. The results show faster, more accurate, and more robust navigation, highlighting the practical impact of memory-guided, open-world VLN for embodied AI tasks.

Abstract

Vision-Language Navigation aims to enable agents to navigate to a target location based on language instructions. Traditional VLN often follows a close-set assumption, i.e., training and test data share the same style of the input images and instructions. However, the real world is open and filled with various unseen environments, posing enormous difficulties for close-set methods. To this end, we focus on the General Scene Adaptation (GSA-VLN) task, aiming to learn generalized navigation ability by introducing diverse environments and inconsistent intructions.Towards this task, when facing unseen environments and instructions, the challenge mainly lies in how to enable the agent to dynamically produce generalized strategies during the navigation process. Recent research indicates that by means of fast and slow cognition systems, human beings could generate stable policies, which strengthen their adaptation for open world. Inspired by this idea, we propose the slow4fast-VLN, establishing a dynamic interactive fast-slow reasoning framework. The fast-reasoning module, an end-to-end strategy network, outputs actions via real-time input. It accumulates execution records in a history repository to build memory. The slow-reasoning module analyze the memories generated by the fast-reasoning module. Through deep reflection, it extracts experiences that enhance the generalization ability of decision-making. These experiences are structurally stored and used to continuously optimize the fast-reasoning module. Unlike traditional methods that treat fast-slow reasoning as independent mechanisms, our framework enables fast-slow interaction. By leveraging the experiences from slow reasoning. This interaction allows the system to continuously adapt and efficiently execute navigation tasks when facing unseen scenarios.

Figures (8)

• Figure 1: In the GSA-VLN task, the training set focuses on residential environments, while the test set includes non-residential scenes such as shopping malls, offices, and cinemas. It also incorporates basic, scene, and user-style instructions. The core goal is to evaluate the agent’s scene generalization ability through diverse building types and instruction variations. To address this open-world navigation challenge, we propose the interactive Slow4Fast framework: “fast reasoning” is driven by a policy network that outputs actions from real-time input and stores memories; “slow reasoning” processes memories, extract generalized experiences, and reinforce the policy network.
• Figure 2: Overview of our method. The policy network processes real-time input, executes actions, and stores historical memory. The slow-reasoning network reflects on these memories to generate generalized experiences, which are then stored. These experiences guide the fast-reasoning network, providing strategic insights when faced with complex scenarios.
• Figure 3: Case Study. The left side shows the execution trajectory of the agent with fast reasoning only, while the right side displays the agent's execution trajectory after slow reasoning optimization. A check mark ($\checkmark$) indicates the destination (next to the door near the floor vent); A five-pointed star ($\star$) marks the final position reached by the agent.
• Figure 4: Predicted trajectories of GR-DUET (left) and our method (right). A check mark ($\checkmark$) indicates the destination; A five-pointed star ($\star$) marks the final position reached by the agent.
• Figure 5: Prompt Template for the Reflection Module.