VISTA: Generative Visual Imagination for Vision-and-Language Navigation

TL;DR

VISTA tackles Vision-and-Language Navigation under long-horizon uncertainty by introducing a closed-loop agent that imagines future goals and grounds them in real-time perception. It combines an Adaptive Imagination Scheduler (AIS), a Perceptual Alignment Filter (PAF), and a Navigational Chain-of-Thought (CoT) reasoning module to produce forward-looking, interpretable navigation decisions, with $z_t = \text{static}$ if $u_t < \tau_u$ and $s_t > \tau_s$, else $z_t = \text{dynamic}$. Experiments on Room-to-Room (R2R) and RoboTHOR show state-of-the-art performance, with improvements in $SR$ and $SPL$ and strong generalization to unseen environments; ablations confirm that imagination, grounding, and CoT each contribute to robustness on long trajectories. The work demonstrates that integrating generative priors with structured reasoning can yield interpretable, ahead-of-need navigation, and provides resources (e.g., R2R-Imagine) to support further research in visually grounded planning for embodied agents.

Abstract

Vision-and-Language Navigation (VLN) tasks agents with locating specific objects in unseen environments using natural language instructions and visual cues. Many existing VLN approaches typically follow an 'observe-and-reason' schema, that is, agents observe the environment and decide on the next action to take based on the visual observations of their surroundings. They often face challenges in long-horizon scenarios due to limitations in immediate observation and vision-language modality gaps. To overcome this, we present VISTA, a novel framework that employs an 'imagine-and-align' navigation strategy. Specifically, we leverage the generative prior of pre-trained diffusion models for dynamic visual imagination conditioned on both local observations and high-level language instructions. A Perceptual Alignment Filter module then grounds these goal imaginations against current observations, guiding an interpretable and structured reasoning process for action selection. Experiments show that VISTA sets new state-of-the-art results on Room-to-Room (R2R) and RoboTHOR benchmarks, e.g.,+3.6% increase in Success Rate on R2R. Extensive ablation analysis underscores the value of integrating forward-looking imagination, perceptual alignment, and structured reasoning for robust navigation in long-horizon environments.

Figures (4)

• Figure 1: Overview of VISTA---At each step, the agent predicts what it expects to see, imagines that goal visually, aligns the prediction with reality, and reasons through language to decide how to move. This loop is guided by an adaptive scheduler that balances global task intent with local observations, enabling dynamic goal prediction. By grounding imagination in perception and embedding structured reasoning into decision-making, VISTA offers a forward-looking and interpretable approach to vision-and-language navigation.
• Figure 2: Chain-of-Thought (CoT) reasoning comparison on a navigation step. Given the same instruction, observation candidates, and history, we compare zero-shot GPT-4, LLaMA2 baseline, and our VISTA agent. While other methods rely on textual cues only, our agent grounds its reasoning in visual imagination and attention alignment. VISTA successfully identifies the correct observation (D) by matching imagined scene with actual perception.
• Figure 3: Comparison on RoboTHOR. (a) Qualitative comparison between VISTA and CoWs. (b) Table comparison of different models.
• Figure 4: Ablation and Planning Analysis. Bar chart and trajectory-level scatter plot illustrating the effect of removing key modules and VISTA’s robustness to path length.