Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Comprehensive Survey on World Models for Embodied AI

Xinqing Li, Xin He, Le Zhang, Min Wu, Xiaoli Li, Yun Liu

TL;DR

This survey formalizes world models for embodied AI using a three-axis taxonomy to unify functionality, temporal modeling, and spatial representation. It surveys data resources, metrics, and quantitative performance across robotics, autonomous driving, and general video domains, highlighting RSSM/Dreamer style sequential models and global-difference approaches like transformer-based and diffusion-based methods. Key insights include long-horizon error accumulation as a core challenge, the trade-off between pixel fidelity and physical consistency, and the fragmentation of datasets and evaluation standards limiting cross-domain generalization. The work advocates for unified, physically grounded benchmarks and hybrid architectures that blend autoregressive and global prediction strengths to advance safe, real-time, and transferable embodied agents.

Abstract

Embodied AI requires agents that perceive, act, and anticipate how actions reshape future world states. World models serve as internal simulators that capture environment dynamics, enabling forward and counterfactual rollouts to support perception, prediction, and decision making. This survey presents a unified framework for world models in embodied AI. Specifically, we formalize the problem setting and learning objectives, and propose a three-axis taxonomy encompassing: (1) Functionality, Decision-Coupled vs. General-Purpose; (2) Temporal Modeling, Sequential Simulation and Inference vs. Global Difference Prediction; (3) Spatial Representation, Global Latent Vector, Token Feature Sequence, Spatial Latent Grid, and Decomposed Rendering Representation. We systematize data resources and metrics across robotics, autonomous driving, and general video settings, covering pixel prediction quality, state-level understanding, and task performance. Furthermore, we offer a quantitative comparison of state-of-the-art models and distill key open challenges, including the scarcity of unified datasets and the need for evaluation metrics that assess physical consistency over pixel fidelity, the trade-off between model performance and the computational efficiency required for real-time control, and the core modeling difficulty of achieving long-horizon temporal consistency while mitigating error accumulation. Finally, we maintain a curated bibliography at https://github.com/Li-Zn-H/AwesomeWorldModels.

A Comprehensive Survey on World Models for Embodied AI

TL;DR

This survey formalizes world models for embodied AI using a three-axis taxonomy to unify functionality, temporal modeling, and spatial representation. It surveys data resources, metrics, and quantitative performance across robotics, autonomous driving, and general video domains, highlighting RSSM/Dreamer style sequential models and global-difference approaches like transformer-based and diffusion-based methods. Key insights include long-horizon error accumulation as a core challenge, the trade-off between pixel fidelity and physical consistency, and the fragmentation of datasets and evaluation standards limiting cross-domain generalization. The work advocates for unified, physically grounded benchmarks and hybrid architectures that blend autoregressive and global prediction strengths to advance safe, real-time, and transferable embodied agents.

Abstract

Embodied AI requires agents that perceive, act, and anticipate how actions reshape future world states. World models serve as internal simulators that capture environment dynamics, enabling forward and counterfactual rollouts to support perception, prediction, and decision making. This survey presents a unified framework for world models in embodied AI. Specifically, we formalize the problem setting and learning objectives, and propose a three-axis taxonomy encompassing: (1) Functionality, Decision-Coupled vs. General-Purpose; (2) Temporal Modeling, Sequential Simulation and Inference vs. Global Difference Prediction; (3) Spatial Representation, Global Latent Vector, Token Feature Sequence, Spatial Latent Grid, and Decomposed Rendering Representation. We systematize data resources and metrics across robotics, autonomous driving, and general video settings, covering pixel prediction quality, state-level understanding, and task performance. Furthermore, we offer a quantitative comparison of state-of-the-art models and distill key open challenges, including the scarcity of unified datasets and the need for evaluation metrics that assess physical consistency over pixel fidelity, the trade-off between model performance and the computational efficiency required for real-time control, and the core modeling difficulty of achieving long-horizon temporal consistency while mitigating error accumulation. Finally, we maintain a curated bibliography at https://github.com/Li-Zn-H/AwesomeWorldModels.
Paper Structure (30 sections, 16 equations, 1 figure, 8 tables)

This paper contains 30 sections, 16 equations, 1 figure, 8 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Core Concepts
  4. Mathematical Formulation of World Models
  5. Taxonomy
  6. Decision-Coupled World Models
  7. Sequential Simulation and Inference
  8. Global Difference Prediction
  9. General-Purpose World Models
  10. Sequential Simulation and Inference
  11. Global Difference Prediction
  12. Data Resources & Metrics
  13. Data Resources
  14. Simulation Platforms
  15. Interactive Benchmarks
  16. ...and 15 more sections

Figures (1)

  • Figure 1: Structure of this survey. The figure classifies world models along three axes and illustrates representative methods for each, providing a unified view of the field. Figure design inspired in part by assran2025vhafner2025masteringlu2025infinicubechen2024drivinggpt.