Say, Dream, and Act: Learning Video World Models for Instruction-Driven Robot Manipulation

TL;DR

The paper addresses the challenge of enabling precise, long-horizon robotic manipulation by predicting how environments evolve and incorporating that foresight into action. It introduces the Dream4manip framework, which integrates a robust video-based world model (Cosmos-Predict2) through domain adaptation and latent adversarial distillation for fast few-step denoising, a length-agnostic imagination mechanism that compresses trajectories into fixed keyframes, and an in-context conditioned action model that grounds imagined futures in real observations. Key contributions include the selection and adaptation of a high-fidelity world model, a diffusion-based distillation strategy with a defined loss ${\mathcal L}_{\mathcal G} = \lambda {\mathcal L}_{adv}^{\mathcal G} + {\mathcal L}_{rec}$ and $\lambda = 0.1$, frame-rate-agnostic trajectory imagination with $n=93$, and an in-context action model that outputs executable actions conditioned on both imagined and real data. Extensive experiments on LIBERO benchmarks and real-world robot setups demonstrate state-of-the-art performance in Embodiment Consistency, Referring Success Rate, Interaction Success Rate, and Task Completion Rate, validating the practical impact of fast, predictive video-conditioned policies for instruction-driven manipulation.

Abstract

Robotic manipulation requires anticipating how the environment evolves in response to actions, yet most existing systems lack this predictive capability, often resulting in errors and inefficiency. While Vision-Language Models (VLMs) provide high-level guidance, they cannot explicitly forecast future states, and existing world models either predict only short horizons or produce spatially inconsistent frames. To address these challenges, we propose a framework for fast and predictive video-conditioned action. Our approach first selects and adapts a robust video generation model to ensure reliable future predictions, then applies adversarial distillation for fast, few-step video generation, and finally trains an action model that leverages both generated videos and real observations to correct spatial errors. Extensive experiments show that our method produces temporally coherent, spatially accurate video predictions that directly support precise manipulation, achieving significant improvements in embodiment consistency, spatial referring ability, and task completion over existing baselines. Codes & Models will be released.

Figures (5)

• Figure 1: (a) The training pipeline for world model distillation. Additional details are provided in \ref{['sec:distill']}. (b) The pipeline of our proposed policy model. Given the current observation and the instruction, the world model first generates imagined future frames. The in-context conditioned action model then produces actions in a closed-loop manner. Each new observation is fed back into the model to generate the next action.
• Figure 2: The structure of the in-context conditioned action model. We use a transformer-based backbone inherited from ACT zhao2023learningfinegrainedbimanualmanipulation for our action model, separated vision encoder is assembled to process videos and observations. The model will output an action chunk for each observation.
• Figure 3: Qualitative results generated by different video generation models for the instruction “Move the blue block into the wooden box.” (a) Output from the Cosmos-predict2 2B model. The model incorrectly replaces the robot gripper with a Robotiq-style two-finger gripper, likely reflecting common embodiments in its training data. (b) Output from the Cosmos-predict2 14B model fine-tuned on DROID. This version predicts multiview videos from left, right, and wrist cameras, but the camera poses still differ from our test setup, leading to incorrect arm scale. (c) Output from the Wan2.2 14B model. Wan struggles to interpret complex instructions: it fails to infer that the robot should first pick up the block before moving it, and instead assumes the robot is already holding the blue block while moving toward the box. (d) Output from the Cosmos-predict2 2B model after domain adaptation. Domain adaptation enables the model to generate future predictions that remain consistent with the robot’s true embodiment.
• Figure 4: Qualitative results on LIBERO. We show world model generation results across different tasks. The red-bordered frame denotes the conditioning frame, while the remaining frames are generated.
• Figure 5: