NavDreamer: Video Models as Zero-Shot 3D Navigators

TL;DR

NavDreamer addresses the problem of generalizing 3D navigation in open-world environments where data collection is expensive and temporal dynamics are essential. It introduces a pipeline that generates predictive video sequences conditioned on language instructions, uses a VLM to score and select trajectories, then decodes executable waypoints with an inverse dynamics model, and finally applies metric-depth scale correction before real-time low-level control. The core contributions are a zero-shot 3D navigation framework based on generative video models, a benchmarking suite spanning five navigation dimensions, and extensive ablations that clarify the impact of high-level video planning on open-world tasks. The findings demonstrate robust zero-shot generalization to unseen objects and environments, validate the approach in real-world experiments, and highlight the need for improvements in latency and precision for high-reactivity tasks. This work offers a scalable path toward open-world robotic navigation using internet-scale video data and outlines concrete directions for real-time optimization and model compression.

Abstract

Previous Vision-Language-Action models face critical limitations in navigation: scarce, diverse data from labor-intensive collection and static representations that fail to capture temporal dynamics and physical laws. We propose NavDreamer, a video-based framework for 3D navigation that leverages generative video models as a universal interface between language instructions and navigation trajectories. Our main hypothesis is that video's ability to encode spatiotemporal information and physical dynamics, combined with internet-scale availability, enables strong zero-shot generalization in navigation. To mitigate the stochasticity of generative predictions, we introduce a sampling-based optimization method that utilizes a VLM for trajectory scoring and selection. An inverse dynamics model is employed to decode executable waypoints from generated video plans for navigation. To systematically evaluate this paradigm in several video model backbones, we introduce a comprehensive benchmark covering object navigation, precise navigation, spatial grounding, language control, and scene reasoning. Extensive experiments demonstrate robust generalization across novel objects and unseen environments, with ablation studies revealing that navigation's high-level decision-making nature makes it particularly suited for video-based planning.

Figures (16)

• Figure 1: System Overview.Left: The pipeline leverages generative video models to transform text instructions and single RGB images into executable trajectories. Right: We evaluate the framework on five navigation task categories.
• Figure 2: Framework of Optimization through Generative Sampling. The model generates $K$ video samples, which are evaluated by Qwen-VL3 on action safety, scene consistency, and task performance. If no valid samples exist ($\sum \phi(V_i) = 0$), human intervention determines whether to re-sample to obtain the optimal video $V^*$.
• Figure 2: Performance Evaluation Across Different Prompting Strategies.
• Figure 3: High-level Waypoint Decoding and Metric Scale Alignment.$\pi^3$ extracts initial waypoints and point clouds from generated RGB videos, while Moge-2 provides metric-scale depth references to resolve scale ambiguity. A global scale factor derived from cross-referencing both models transforms synthesized trajectories into physically grounded waypoint lists.
• Figure 4: Visualization and Quantitative Analysis of Scale Correction. (A) Reference ground-truth point cloud captured from the real-world environment. (B) Comparative visualization of reconstructed point clouds before and after metric scale alignment. (C) The error comparison of the methods. Note: The relative picture size of (A) and (B) are rendered to be the same as their true physical proportions.