Learning to Retrieve Navigable Candidates for Efficient Vision-and-Language Navigation

TL;DR

Experimental results demonstrate consistent improvements in Success Rate, Oracle Success Rate, and SPL on both seen and unseen environments and indicate that retrieval-augmented decision support is an effective and scalable strategy for enhancing LLM-based vision-and-language navigation.

Abstract

Vision-and-Language Navigation (VLN) requires an agent to follow natural-language instructions and navigate through previously unseen environments. Recent approaches increasingly employ large language models (LLMs) as high-level navigators due to their flexibility and reasoning capability. However, prompt-based LLM navigation often suffers from inefficient decision-making, as the model must repeatedly interpret instructions from scratch and reason over noisy and verbose navigable candidates at each step. In this paper, we propose a retrieval-augmented framework to improve the efficiency and stability of LLM-based VLN without modifying or fine-tuning the underlying language model. Our approach introduces retrieval at two complementary levels. At the episode level, an instruction-level embedding retriever selects semantically similar successful navigation trajectories as in-context exemplars, providing task-specific priors for instruction grounding. At the step level, an imitation-learned candidate retriever prunes irrelevant navigable directions before LLM inference, reducing action ambiguity and prompt complexity. Both retrieval modules are lightweight, modular, and trained independently of the LLM. We evaluate our method on the Room-to-Room (R2R) benchmark. Experimental results demonstrate consistent improvements in Success Rate, Oracle Success Rate, and SPL on both seen and unseen environments. Ablation studies further show that instruction-level exemplar retrieval and candidate pruning contribute complementary benefits to global guidance and step-wise decision efficiency. These results indicate that retrieval-augmented decision support is an effective and scalable strategy for enhancing LLM-based vision-and-language navigation.

Figures (3)

• Figure 1: The agent follows a language-centric pipeline. At each step, a candidate retriever prunes directional observations before LLM reasoning, while an embedding-based instruction retriever retrieves exemplar trajectories at the episode level for in-context guidance. A history summarizer maintains a compact navigation context. The prompt aggregates the system principles, instruction, retrieved exemplars, pruned observations, and history, enabling the LLM to generate reasoning traces and navigation actions in a closed-loop manner.
• Figure 2: Here is a case study of a successful navigation step. At this step, the candidate retriever selects the top 5 directional observations for LLM reasoning. The direction highlighted in blue corresponds to the correct navigable direction chosen by the LLM, while the red directions indicate irrelevant candidates pruned by the candidate retriever. This pruning reduces decision noise and enables the LLM to focus on instruction-consistent actions.
• Figure 3: SR, OSR, and SPL of the baseline agent with different ablation modules on the R2R val-unseen split.