VENTURA: Adapting Image Diffusion Models for Unified Task Conditioned Navigation

TL;DR

Ventura tackles open-world, goal-conditioned navigation by repurposing pre-trained image diffusion models to plan in image space. A diffusion-based planner generates a visual plan $y_{plan}$ conditioned on a robot observation $o$ and a language instruction $\mathcal{L}$, which is then grounded by a lightweight policy to a sequence of xyz waypoints; an auto-labeling pipeline using Co-Tracker enables scalable supervision from uncalibrated video with language captions. Key contributions include a simple finetuning protocol for diffusion planners, an auto-labeling pipeline to produce image-space plans without manual pixel annotations, and an open-source, language-captioned navigation dataset. Empirical results show Ventura outperforming state-of-the-art baselines on diverse tasks, with significant improvements in success rates and collision reduction in both seen and unseen environments, and evidence of emergent compositional capabilities across tasks. The approach offers a scalable path to leveraging internet-scale priors for adaptive, open-world robotic autonomy.

Abstract

Robots must adapt to diverse human instructions and operate safely in unstructured, open-world environments. Recent Vision-Language models (VLMs) offer strong priors for grounding language and perception, but remain difficult to steer for navigation due to differences in action spaces and pretraining objectives that hamper transferability to robotics tasks. Towards addressing this, we introduce VENTURA, a vision-language navigation system that finetunes internet-pretrained image diffusion models for path planning. Instead of directly predicting low-level actions, VENTURA generates a path mask (i.e. a visual plan) in image space that captures fine-grained, context-aware navigation behaviors. A lightweight behavior-cloning policy grounds these visual plans into executable trajectories, yielding an interface that follows natural language instructions to generate diverse robot behaviors. To scale training, we supervise on path masks derived from self-supervised tracking models paired with VLM-augmented captions, avoiding manual pixel-level annotation or highly engineered data collection setups. In extensive real-world evaluations, VENTURA outperforms state-of-the-art foundation model baselines on object reaching, obstacle avoidance, and terrain preference tasks, improving success rates by 33% and reducing collisions by 54% across both seen and unseen scenarios. Notably, we find that VENTURA generalizes to unseen combinations of distinct tasks, revealing emergent compositional capabilities. Videos, code, and additional materials: https://venturapath.github.io

Figures (6)

• Figure 1: Given an image and a language instruction (e.g. avoid the safety vest), Ventura uses a fine-tuned image diffusion model to render a path mask in the image space. The path mask is then passed to a lightweight policy network to produce executable robot actions. By training on a mix of goal-agnostic and goal-directed demonstrations, Ventura grounds diverse language instructions in safe and precise robot motions.
• Figure 2: Ventura Architecture Overview. Our model is composed from an image diffusion planner $\pi_\text{plan}$ and grounding policy $\pi_\text{gnd}$. $\pi_\text{plan}$ generates a "visual plan" in the form of a path mask, guiding $\pi_\text{gnd}$ to generate a sequence of xyz waypoints that satisfy the goal instruction $\mathcal{L}$.
• Figure 3: Ventura Data Generation Pipeline. We generate ground truth path masks by using an off-the-shelf tracker karaev2024cotracker3 to track points traversed by the robot in reversed videos. Each dots' color corresponds to the first time that it is tracked from. We provide seed captions to a VLM to generate diverse language goals that explain the path.
• Figure 4: Ground truth language and path mask labels from the Ventura dataset. We co-train Ventura on a collection of navigation demonstrations with and without language captions. To bolster generalization to novel language prompts, we augment human-labeled captions using a pre-trained VLM to automatically generate diverse caption variations.
• Figure 5: Qualitative analysis of various visual-language navigation baselines.Ventura consistently outperforms existing approaches in terms of task alignment and generalization to unseen entities and environments. We use the following abbreviations: Ventura-P: Our model without initializing with internet-pretrained weights.