Run, Ruminate, and Regulate: A Dual-process Thinking System for Vision-and-Language Navigation

TL;DR

This paper addresses vision-language navigation (VLN) by integrating a fast, domain-specific Runner with a slow, deliberative Ruminator under a Regulator, forming a dual-process framework $R^3$ that operates in a zero-shot setting. The Runner uses a compact transformer-based VLN expert with memory, while the Ruminator relies on GPT-4o with chain-of-thought prompting to perform multi-step reasoning, guided by a two-stage Regulator that detects anomalies via looping and a learned GNN-based scoring, then formulates corrective plans with $P_t$. Across R2R and REVERIE benchmarks, $R^3$ achieves notable gains in SPL and RGSPL (e.g., improvements of 3.28% and 3.30% on REVERIE unseen) and also reduces inference time compared with other LLM-assisted methods, demonstrating both accuracy and efficiency. The approach highlights the practical potential of dynamically switching between fast domain expertise and powerful LLM reasoning to tackle complex VLN tasks in real-world settings.

Abstract

Vision-and-Language Navigation (VLN) requires an agent to dynamically explore complex 3D environments following human instructions. Recent research underscores the potential of harnessing large language models (LLMs) for VLN, given their commonsense knowledge and general reasoning capabilities. Despite their strengths, a substantial gap in task completion performance persists between LLM-based approaches and domain experts, as LLMs inherently struggle to comprehend real-world spatial correlations precisely. Additionally, introducing LLMs is accompanied with substantial computational cost and inference latency. To address these issues, we propose a novel dual-process thinking framework dubbed R3, integrating LLMs' generalization capabilities with VLN-specific expertise in a zero-shot manner. The framework comprises three core modules: Runner, Ruminator, and Regulator. The Runner is a lightweight transformer-based expert model that ensures efficient and accurate navigation under regular circumstances. The Ruminator employs a powerful multimodal LLM as the backbone and adopts chain-of-thought (CoT) prompting to elicit structured reasoning. The Regulator monitors the navigation progress and controls the appropriate thinking mode according to three criteria, integrating Runner and Ruminator harmoniously. Experimental results illustrate that R3 significantly outperforms other state-of-the-art methods, exceeding 3.28% and 3.30% in SPL and RGSPL respectively on the REVERIE benchmark. This pronounced enhancement highlights the effectiveness of our method in handling challenging VLN tasks.

Figures (5)

• Figure 1: Comparison of inference efficiency and navigation performance. Our R$^3$ requires only one-fifth of the inference time compared with other LLM-assisted methods. NavGPT-2 exhibits a little better efficiency since it deploys LLMs locally while others query the GPT model via API.
• Figure 2: Overview of the proposed R$^3$. Our system comprises three core modules: Runner, Ruminator, and Regulator. The working flow operates as (i)$\rightarrow$(ii)$\rightarrow$(iii). The navigation initiates with the Runner. For each timestep, the Regulator evaluates the current condition. If the condition is nominal, the Runner proceeds ((iii)-a); otherwise, the Ruminator engages to resolve exceptions ((iii)-b)
• Figure 3: Overall pipeline of the proposed R$^3$. For each timestep $t$, the Regulator evaluates the navigation condition and switches to Ruminator if necessary, which resorts to mLLMs for resolving the anomalies; otherwise, the Runner, a lightweight, transformer-based VLN expert, proceeds with navigation efficiently. Here $V_t$ represents all inputs, including history $H_t$, instruction $I$, RGB-D images $O_t$, $D_t$, and pose $R_t$.
• Figure 4: Textual template of inputs. By systematically formalizing the inputs, the Ruminator is capable of extracting current navigational circumstance in a more explicit and effective manner.
• Figure 5: Representative qualitative results on REVERIE validation unseen split. Benefiting from long-term planning capabilities, our approach can effectively escape the wandering anomaly and successfully complete the episode.