DreamGen: Unlocking Generalization in Robot Learning through Video World Models

TL;DR

DreamGen tackles the data collection bottleneck in robot learning by turning video world models into synthetic data generators. It fine-tunes models on a target robot, generates language-conditioned video rollouts, and extracts pseudo-actions via IDM or latent-action models to train visuomotor policies on neural trajectories. The approach enables substantial generalization across unseen behaviors and environments, demonstrated on multiple embodiments and tasks, and is complemented by DreamGen Bench to assess world-model adaptation. This work suggests that scalable robot learning is achievable with powerful video priors, reducing manual teleoperation requirements.

Abstract

We introduce DreamGen, a simple yet highly effective 4-stage pipeline for training robot policies that generalize across behaviors and environments through neural trajectories - synthetic robot data generated from video world models. DreamGen leverages state-of-the-art image-to-video generative models, adapting them to the target robot embodiment to produce photorealistic synthetic videos of familiar or novel tasks in diverse environments. Since these models generate only videos, we recover pseudo-action sequences using either a latent action model or an inverse-dynamics model (IDM). Despite its simplicity, DreamGen unlocks strong behavior and environment generalization: a humanoid robot can perform 22 new behaviors in both seen and unseen environments, while requiring teleoperation data from only a single pick-and-place task in one environment. To evaluate the pipeline systematically, we introduce DreamGen Bench, a video generation benchmark that shows a strong correlation between benchmark performance and downstream policy success. Our work establishes a promising new axis for scaling robot learning well beyond manual data collection. Code available at https://github.com/NVIDIA/GR00T-Dreams.

Figures (12)

• Figure 1: Generalization through DreamGen. We enable 2D visuomotor robot policies to generalize to new environments with new behaviors, while only collecting teleoperation data for a single behavior type (pick&place) in a single environment by utilizing video world models as synthetic data generators.
• Figure 2: DreamGen Overview. We begin by fine-tuning a video world model on teleoperated robot trajectories. Given an initial frame and a language instruction, the model generates video rollouts depicting the intended behavior. As these videos lack action annotations, we infer pseudo-actions using either a latent action model or an inverse dynamics model, forming what we call neural trajectories. Finally, we train visuomotor robot policies on these neural trajectories.
• Figure 3: Extracting Pseudo Actions. (a) shows the architecture of our IDM model and (b) shows the architecture of our latent action model.
• Figure 4: Scaling # of Neural Trajectories in RoboCasa. We vary the sizes of neural trajectories (x-axis) and ground-truth trajectories (low, mid, high) and report results with both latent and IDM actions as pseudo action labels. We report the average success rate (%) across 24 tasks. The results at $x=0$ correspond to the baseline only trained on ground-truth videos.
• Figure 5: Real-world Robot Evaluation Results. The red rectangular box shows the range of object randomization during training and evaluation. Low Data denotes training 10% of available training data (only 10 trajectories per task except for GR1-folding, where we used 25 trajectories), and Low Data + Neural Traj. denotes co-training with neural trajectories.