World Action Models are Zero-shot Policies

TL;DR

DreamZero introduces DreamZero, a 14B World Action Model that jointly predicts future video frames and robot actions from language and observations, leveraging a pretrained video diffusion backbone to encode rich spatiotemporal priors. By learning dynamics directly from video and action in a unified model, it achieves zero-shot generalization to unseen tasks and environments, strong cross-embodiment transfer using video-only demonstrations, and few-shot adaptation to new embodiments. System-, implementation-, and model-level optimizations yield a 38× inference speedup, enabling real-time closed-loop control at 7 Hz, and DreamZero-Flash further reduces inference steps with minimal loss in performance. The approach demonstrates that WAMs can outperform Vision-Language-Action models on generalization benchmarks, retain generalization after post-training, and enable data-efficient transfer across robots and even humans, while open-sourcing weights and code to promote reproducibility. These results suggest a scalable pathway for robot foundation models that leverage large-scale video priors and diverse data to achieve broad, adaptable manipulation capabilities.

Abstract

State-of-the-art Vision-Language-Action (VLA) models excel at semantic generalization but struggle to generalize to unseen physical motions in novel environments. We introduce DreamZero, a World Action Model (WAM) built upon a pretrained video diffusion backbone. Unlike VLAs, WAMs learn physical dynamics by predicting future world states and actions, using video as a dense representation of how the world evolves. By jointly modeling video and action, DreamZero learns diverse skills effectively from heterogeneous robot data without relying on repetitive demonstrations. This results in over 2x improvement in generalization to new tasks and environments compared to state-of-the-art VLAs in real robot experiments. Crucially, through model and system optimizations, we enable a 14B autoregressive video diffusion model to perform real-time closed-loop control at 7Hz. Finally, we demonstrate two forms of cross-embodiment transfer: video-only demonstrations from other robots or humans yield a relative improvement of over 42% on unseen task performance with just 10-20 minutes of data. More surprisingly, DreamZero enables few-shot embodiment adaptation, transferring to a new embodiment with only 30 minutes of play data while retaining zero-shot generalization.

Figures (17)

• Figure 1: Overview. By jointly predicting video and action, World Action Models (WAMs) inherit world physics priors that enable 1) effective learning from diverse, non-repetitive data, 2) open-world generalization, 3) cross-embodiment learning from video-only data, and 4) few-shot adaptation to new robots.
• Figure 2: Joint Video and Action Prediction. DreamZero jointly generates video and action. We observe that the predicted actions closely align with the generated video. The examples are from totally unseen tasks.
• Figure 3: Free-form Evaluation.DreamZero performs a diverse range of tasks when conditioned on natural language instructions, including object manipulation, tool use, and human-robot interaction.
• Figure 4: Model Architecture of DreamZero. The model takes three inputs: visual context (encoded via a VAE), language instructions (via a text encoder), and proprioceptive state (via a state encoder). These are processed by an autoregressive DiT backbone using flow matching, which jointly predicts future video frames and actions through separate decoders. During training (left), for each chunk, the model denoises noisy video and action latents conditioned on clean video context. During inference (right), predictions are executed asynchronously in the real world, and ground-truth observations are fed back into the KV cache to prevent error accumulation.
• Figure 5: Decoupled Noise Schedules.DreamZero (blue) uses coupled noise for video and action (both uniform). DreamZero-Flash (red) biases video toward high-noise states via a Beta distribution while keeping action noise uniform, training the model to predict clean actions from noisy visual context.