Video Generation Models in Robotics -- Applications, Research Challenges, Future Directions

TL;DR

Video Generation Models in Robotics surveys the use of high-fidelity video generators as embodied world models for robotics, contrasting Markovian state-based approaches with video-based dynamics and highlighting diffusion/flow-matching and video-JEPA as leading paradigms. It details how these models enable data generation for imitation learning, dynamics and reward modeling for RL, scalable policy evaluation, and visual planning, while placing emphasis on challenges such as hallucinations, physics violations, uncertainty quantification, instruction following, and substantial data/training/inference costs. The paper reviews model architectures (latent diffusion, DiT, U-Nets, VQ-VAE), conditioning modalities (text, images, trajectories), and evaluation benchmarks, and it articulates future directions including physics-informed priors, safety guardrails, long-horizon generation, and robotics-centric benchmarks. Overall, it argues that high-fidelity, controllable video world models can substantially accelerate safe, scalable robot policy learning and evaluation, provided advances in reliability, safety, and data efficiency are achieved.

Abstract

Video generation models have emerged as high-fidelity models of the physical world, capable of synthesizing high-quality videos capturing fine-grained interactions between agents and their environments conditioned on multi-modal user inputs. Their impressive capabilities address many of the long-standing challenges faced by physics-based simulators, driving broad adoption in many problem domains, e.g., robotics. For example, video models enable photorealistic, physically consistent deformable-body simulation without making prohibitive simplifying assumptions, which is a major bottleneck in physics-based simulation. Moreover, video models can serve as foundation world models that capture the dynamics of the world in a fine-grained and expressive way. They thus overcome the limited expressiveness of language-only abstractions in describing intricate physical interactions. In this survey, we provide a review of video models and their applications as embodied world models in robotics, encompassing cost-effective data generation and action prediction in imitation learning, dynamics and rewards modeling in reinforcement learning, visual planning, and policy evaluation. Further, we highlight important challenges hindering the trustworthy integration of video models in robotics, which include poor instruction following, hallucinations such as violations of physics, and unsafe content generation, in addition to fundamental limitations such as significant data curation, training, and inference costs. We present potential future directions to address these open research challenges to motivate research and ultimately facilitate broader applications, especially in safety-critical settings.

Figures (6)

• Figure 1: Overview. As embodied world models, video models generate high-fidelity predictions of the spatiotemporal evolution of real-world environments, capturing fine-grained robot-environment interactions that have been traditionally challenging for classical physics-based simulators. Their remarkable capabilities enable generalist robot policy learning, policy evaluation, and visual planning that is well-aligned with commonsense knowledge.
• Figure 2: Organization. The organization of this survey, including background material, taxonomy of robotics applications, evaluation metrics and benchmarks, and open challenges and directions for future research.
• Figure 3: Diffusion Video Model Architectures. Diffusion/Flow-matching has emerged as the dominant model architecture for training photorealistic controllable video models that can be steered using text, image, and other conditioning inputs. These models broadly utilize diffusion transformers (DiTs) or U-Nets to learn important interpendencies across space and time within a compact latent space.
• Figure 4: Video Models for Embodied World Modeling. Video models provide high-quality representations of the physical world, which could be implicit (e.g., latent and video representations) or explicit (e.g., point clouds and Gaussian Splatting models).
• Figure 5: Video Models for Data Generation. Video models enable high-fidelity data generation for cost-effective policy learning. Robot actions can be extracted from videos through modular approaches using end-effector pose tracking or end-to-end approaches, such as inverse-dynamics methods.