FantasyVLN: Unified Multimodal Chain-of-Thought Reasoning for Vision-Language Navigation

TL;DR

Vision-Language Navigation requires robust multimodal understanding and long-horizon planning. FantasyVLN introduces a unified, implicit multimodal Chain-of-Thought framework that compresses imagined visuals into a latent VAR space and trains across textual, visual, and multimodal CoT modes, enabling real-time inference without explicit CoT generation. A cross-mode alignment constraint and data-mixture training ensure modality-invariant reasoning representations, yielding substantial gains in navigation accuracy and efficiency on LH-VLN compared to explicit CoT baselines. The approach demonstrates the practical viability of latent, multimodal reasoning for embodied agents in complex environments, with reduced latency and improved generalization. Overall, this work provides a scalable path toward real-time, reasoning-aware VLN via latent, cross-modal supervision and implicit CoT representations.

Abstract

Achieving human-level performance in Vision-and-Language Navigation (VLN) requires an embodied agent to jointly understand multimodal instructions and visual-spatial context while reasoning over long action sequences. Recent works, such as NavCoT and NavGPT-2, demonstrate the potential of Chain-of-Thought (CoT) reasoning for improving interpretability and long-horizon planning. Moreover, multimodal extensions like OctoNav-R1 and CoT-VLA further validate CoT as a promising pathway toward human-like navigation reasoning. However, existing approaches face critical drawbacks: purely textual CoTs lack spatial grounding and easily overfit to sparse annotated reasoning steps, while multimodal CoTs incur severe token inflation by generating imagined visual observations, making real-time navigation impractical. In this work, we propose FantasyVLN, a unified implicit reasoning framework that preserves the benefits of CoT reasoning without explicit token overhead. Specifically, imagined visual tokens are encoded into a compact latent space using a pretrained Visual AutoRegressor (VAR) during CoT reasoning training, and the model jointly learns from textual, visual, and multimodal CoT modes under a unified multi-CoT strategy. At inference, our model performs direct instruction-to-action mapping while still enjoying reasoning-aware representations. Extensive experiments on LH-VLN show that our approach achieves reasoning-aware yet real-time navigation, improving success rates and efficiency while reducing inference latency by an order of magnitude compared to explicit CoT methods.

Figures (5)

• Figure 1: Overview of FantasyVLN. FantasyVLN is a VLN framework that integrates the strengths of textual and visual CoT reasoning modes, thereby jointly modeling semantic planning and spatial understanding.
• Figure 2: Overview of our unified multimodal Chain-of-Thought reasoning framework. The model supports four reasoning modes under a shared architecture: (a) non-CoT reasoning for real-time inference, (b) textual CoT, (c) visual CoT enabled by VAR-compressed imagined observations, and (d) multimodal CoT combining textual and visual reasoning. A gating mechanism switches the model across reasoning modes, while the action predictions from CoT modes are consistently aligned with the non-CoT mode.
• Figure 3: ISR variation with respect to different VAR scales.
• Figure 4: Qualitative comparison of image reconstruction results produced by the VAR model using latent inputs across different scales. For each image, the VAR model receives the ground truth latents up to a specified scale and predicts all remaining scales; the final reconstruction is obtained by decoding the combined ground truth and predicted latents.
• Figure 5: Comparison of training efficiency between FantasyVLN and WorldVLA.