DreamToNav: Generalizable Navigation for Robots via Generative Video Planning

TL;DR

Results demonstrate that trajectories extracted from generative video predictions can be reliably executed on physical robots across different locomotion platforms.

Abstract

We present DreamToNav, a novel autonomous robot framework that uses generative video models to enable intuitive, human-in-the-loop control. Instead of relying on rigid waypoint navigation, users provide natural language prompts (e.g. ``Follow the person carefully''), which the system translates into executable motion. Our pipeline first employs Qwen 2.5-VL-7B-Instruct to refine vague user instructions into precise visual descriptions. These descriptions condition NVIDIA Cosmos 2.5, a state-of-the-art video foundation model, to synthesize a physically consistent video sequence of the robot performing the task. From this synthetic video, we extract a valid kinematic path using visual pose estimation, robot detection and trajectory recovery. By treating video generation as a planning engine, DreamToNav allows robots to visually "dream" complex behaviors before executing them, providing a unified framework for obstacle avoidance and goal-directed navigation without task-specific engineering. We evaluate the approach on both a wheeled mobile robot and a quadruped robot in indoor navigation tasks. DreamToNav achieves a success rate of 76.7%, with final goal errors typically within 0.05-0.10 m and trajectory tracking errors below 0.15 m. These results demonstrate that trajectories extracted from generative video predictions can be reliably executed on physical robots across different locomotion platforms.

Figures (7)

• Figure 1: Pipeline for trajectory extraction from a single image and text prompt. A user provides an input image of a robot and a textual prompt describing the desired motion. The image prompt pair is processed by a VLM and the COSMOS video generation module to synthesize a plausible motion sequence. Visual odometry and pose estimation are then applied to the generated video to recover robot poses and track visual features over time. The final output is the estimated trajectory of the robot derived solely from the initial image and text prompt.
• Figure 2: Robot detection results using the trained YOLO11n model. The detector identifies both a wheeled UGV platform and a quadruped robot dog on diffused and real frames.
• Figure 3: Robot pose estimation pipeline. Left: robot detection in the camera image. Right: estimated robot trajectory obtained from visual odometry and pose estimation compared with the ground-truth robot pose recorded by the VICON motion capture system.
• Figure 4: Trajectory extraction from generated video frames. The robot is detected in each generated frame (right), and its pose is estimated using PnP to recover the robot trajectory (blue). The corresponding camera trajectory obtained from visual odometry is shown in red.
• Figure 5: Experimental evaluation of the proposed navigation framework. A user first captures an initial frame of the environment. Based on this image, two navigation tasks are generated: (1) move to the red square object and (2) move to the blue square object. The figure shows the initial frame, the final generated frames corresponding to each task, and the resulting UGV trajectories. The red and blue dots represent the robot pose estimated using visual odometry during execution, while the black dashed trajectory corresponds to the ground-truth robot motion recorded by the VICON motion capture system. The axes xy indicate the camera pose of the initial captured frame.