A Step Toward World Models: A Survey on Robotic Manipulation

TL;DR

This survey frames world models as essential internal representations that enable robotic manipulation by predicting dynamics, planning actions, and guiding learning. It categorizes world-model approaches into implicit, latent-dynamics, and video-generation paradigms, and analyzes their architectures (flat vs hierarchical), representations (scene/object/flow-centric), and task scopes (single vs multi-task). It highlights core functions (decision support and training facilitation) and surveys key techniques, challenges, and datasets, while proposing a capability framework comprising perception, imagination, long-horizon reasoning, memory, and causality. The paper argues for integrating modern foundation-models, multi-modal sensing, and physics-informed learning to realize generalizable, practical world models for robotics, and outlines future directions including data, computation, evaluation standards, and safety considerations. Overall, it provides a comprehensive, problem-centered roadmap toward robust, scalable world modeling for embodied agents in manipulation tasks.

Abstract

Autonomous agents are increasingly expected to operate in complex, dynamic, and uncertain environments, performing tasks such as manipulation, navigation, and decision-making. Achieving these capabilities requires agents to understand the underlying mechanisms and dynamics of the world, moving beyond reactive control or simple replication of observed states. This motivates the development of world models as internal representations that encode environmental states, capture dynamics, and support prediction, planning, and reasoning. Despite growing interest, the definition, scope, architectures, and essential capabilities of world models remain ambiguous. In this survey, we go beyond prescribing a fixed definition and limiting our scope to methods explicitly labeled as world models. Instead, we examine approaches that exhibit the core capabilities of world models through a review of methods in robotic manipulation. We analyze their roles across perception, prediction, and control, identify key challenges and solutions, and distill the core components, capabilities, and functions that a fully realized world model should possess. Building on this analysis, we aim to motivate further development toward generalizable and practical world models for robotics.

Figures (4)

• Figure 1: Conceptual flow of the survey. The survey aims to clarify the motivation behind world modeling, explore its essential scope and development, and illuminate pathways toward more general and capable world models.
• Figure 2: A visualization of an agent lecun2022path, where the world model predicts possible future world states as a function of imagined actions sequences proposed by the actor.
• Figure 3: An overview of world models. Implicit world models map observations and instructions directly to actions without explicitly modeling environmental dynamics. Latent-dynamics world models capture the evolution of environment dynamics within a latent space, while video-generation-based world models predict future visual scenes.
• Figure 4: Potential Core Components and Capabilities of World Models.