Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ImagiNav: Scalable Embodied Navigation via Generative Visual Prediction and Inverse Dynamics

Jie Chen, Yuxin Cai, Yizhuo Wang, Ruofei Bai, Yuhong Cao, Jun Li, Yau Wei Yun, Guillaume Sartoretti

Abstract

Enabling robots to navigate open-world environments via natural language is critical for general-purpose autonomy. Yet, Vision-Language Navigation has relied on end-to-end policies trained on expensive, embodiment-specific robot data. While recent foundation models trained on vast simulation data show promise, the challenge of scaling and generalizing due to the limited scene diversity and visual fidelity in simulation persists. To address this gap, we propose ImagiNav, a novel modular paradigm that decouples visual planning from robot actuation, enabling the direct utilization of diverse in-the-wild navigation videos. Our framework operates as a hierarchy: a Vision-Language Model first decomposes instructions into textual subgoals; a finetuned generative video model then imagines the future video trajectory towards that subgoal; finally, an inverse dynamics model extracts the trajectory from the imagined video, which can then be tracked via a low-level controller. We additionally develop a scalable data pipeline of in-the-wild navigation videos auto-labeled via inverse dynamics and a pretrained Vision-Language Model. ImagiNav demonstrates strong zero-shot transfer to robot navigation without requiring robot demonstrations, paving the way for generalist robots that learn navigation directly from unlabeled, open-world data.

ImagiNav: Scalable Embodied Navigation via Generative Visual Prediction and Inverse Dynamics

Abstract

Enabling robots to navigate open-world environments via natural language is critical for general-purpose autonomy. Yet, Vision-Language Navigation has relied on end-to-end policies trained on expensive, embodiment-specific robot data. While recent foundation models trained on vast simulation data show promise, the challenge of scaling and generalizing due to the limited scene diversity and visual fidelity in simulation persists. To address this gap, we propose ImagiNav, a novel modular paradigm that decouples visual planning from robot actuation, enabling the direct utilization of diverse in-the-wild navigation videos. Our framework operates as a hierarchy: a Vision-Language Model first decomposes instructions into textual subgoals; a finetuned generative video model then imagines the future video trajectory towards that subgoal; finally, an inverse dynamics model extracts the trajectory from the imagined video, which can then be tracked via a low-level controller. We additionally develop a scalable data pipeline of in-the-wild navigation videos auto-labeled via inverse dynamics and a pretrained Vision-Language Model. ImagiNav demonstrates strong zero-shot transfer to robot navigation without requiring robot demonstrations, paving the way for generalist robots that learn navigation directly from unlabeled, open-world data.
Paper Structure (22 sections, 8 equations, 5 figures, 2 tables)

This paper contains 22 sections, 8 equations, 5 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Vision-Language Navigation
  4. Generative Models
  5. Inverse Dynamics from Visual Observations
  6. ImagiNav
  7. Semantic Reasoning
  8. Visual Imagination
  9. Geometry Inverse Dynamics
  10. Scalable Data Collection Pipeline
  11. Auto-Labeling via IDM-Guided Annotation
  12. Geometric Motion Primitive Extraction
  13. Motion-Primitive–Guided Captioning
  14. Experiments
  15. Simulation Experimental Setup
  16. ...and 7 more sections

Figures (5)

  • Figure 1: Overview of the ImagiNav Framework. Top: Navigation is formulated in visual space. A generative visual planner synthesizes future egocentric observations, which are geometrically grounded into pose trajectories via inverse dynamics and executed by a tracking controller. Bottom: Geometry-first scalable training pipeline from in-the-wild egocentric videos.
  • Figure 2: Dataset Statistics.(a) Distribution of raw videos across different real-world environments. (b) Distribution of motion primitives in the final augmented dataset, showing balanced coverage.
  • Figure 3: Qualitative Visualization of the Imagination-Action Cycle.(Left) The imagined future frame generated by the model. (Middle) The actual robot observation after executing the extracted plan. (Right) Top-down trajectory map, where the red line indicates the executed path and the green line represents the reference ground truth. The close alignment confirms the physical consistency of the generated plans.
  • Figure 4: Qualitative Comparison of Simulation vs. Real-World Training. We visualize the generated future trajectories across Ground Truth (Top Row), ImagiNav-Real (Middle Row), and ImagiNav-Sim (Bottom Row). Real-world data demonstrates superior understanding of static geometry (Scene A, the navigable walkway is circled in green), and dynamic behavior (Scene B).
  • Figure 5: Controllability and Geometric Grounding. Given a single start frame, the model generates diverse trajectories conditioned on different language instructions (from left to right: "Move forward and turn left", "Move forward", and "Move forward and turn right").