MapDream: Task-Driven Map Learning for Vision-Language Navigation

TL;DR

VLN faces partial observability and relies on spatial context to ground instructions. MapDream rethinks maps as task-driven, autoregressively generated BEV representations $M_t = G(O_t, o_t, I)$ that are tightly integrated with a VLN policy in a map-in-the-loop framework. A two-stage training regimen — supervised pre-training with task-driven BEV signals and reinforcement fine-tuning with GRPO — aligns maps with navigation objectives, yielding state-of-the-art monocular performance on R2R-CE and RxR-CE and strong generalization to unseen environments. Real-world demonstrations on a monocular setup show zero-shot transfer to indoor settings, underscoring the practical impact of learning spatial abstractions directly from navigation objectives and instructions.

Abstract

Vision-Language Navigation (VLN) requires agents to follow natural language instructions in partially observed 3D environments, motivating map representations that aggregate spatial context beyond local perception. However, most existing approaches rely on hand-crafted maps constructed independently of the navigation policy. We argue that maps should instead be learned representations shaped directly by navigation objectives rather than exhaustive reconstructions. Based on this insight, we propose MapDream, a map-in-the-loop framework that formulates map construction as autoregressive bird's-eye-view (BEV) image synthesis. The framework jointly learns map generation and action prediction, distilling environmental context into a compact three-channel BEV map that preserves only navigation-critical affordances. Supervised pre-training bootstraps a reliable mapping-to-control interface, while the autoregressive design enables end-to-end joint optimization through reinforcement fine-tuning. Experiments on R2R-CE and RxR-CE achieve state-of-the-art monocular performance, validating task-driven generative map learning.

Figures (4)

• Figure 1: Map-in-the-Loop Architecture. Unlike previous approaches that either omit maps or rely on expert-designed representations, MapDream adopts a map-in-the-loop design that learns a task-driven generative map jointly with the navigation policy. Red dashed arrows denote training-time gradient flow from navigation objectives, illustrating how the learned map representation is directly shaped by downstream tasks. Abbreviations: Obs denotes observations, Inst instructions, and Act actions.
• Figure 2: Overview of the MapDream Framework. The diagram shows the two-stage optimization of MapDream. Stage 1 learns structured task-driven maps from visual observations and language instructions for initialization and supervised policy training. Stage 2 jointly optimizes the map module and VLN policy through reinforcement learning under a unified navigation objective, allowing the map to be shaped by downstream tasks.
• Figure 3: Qualitative navigation example illustrating the effect of task-driven maps in MapDream. (Left) Trajectory comparison shows that MapDream (green) closely follows the ground-truth path (blue), while the VLN policy without maps deviates (red). (Right) Conditioned on monocular observations, MapDream generates task-driven BEV maps that retain navigation-critical spatial cues such as occupancy, distance, and landmarks. These maps provide compact task-driven abstractions for decision-making. The yellow-highlighted region marks a spatial decision point where the no-map policy deviates, whereas MapDream selects the correct path using map information.
• Figure 4: Real-World Navigation with Task-Driven Maps. Real-world deployment of MapDream on a humanoid platform. Given monocular input and a natural language instruction, MapDream constructs robot-centric task-driven BEV maps from the forward-facing viewpoint. The maps capture navigation-relevant spatial affordances over time and guide the robot to execute accurate long-horizon navigation in real indoor environments.