World Models as an Intermediary between Agents and the Real World

TL;DR

This paper tackles the bottleneck of achieving superhuman performance in high-cost domains by decoupling agent learning from real-world execution through data-driven world models. It defines a unified world-model framework $\\mathcal{W} = (\\hat{T}, \\hat{R}, \\hat{G})$ that learns dynamics, rewards, and task distributions from offline data, enabling low-cost planning and evaluation via rollouts in the learned simulator and selective real-world data collection. The authors argue that existing strategies (supervised scaling, offline RL, and simulators) are insufficient to address cross-task generalization, safety, and sample efficiency, and present a broad, domain-spanning set of applications (robotics, service agents, computer use, science) with concrete mechanisms like OOD task generation, dense feedback, time-compressed dynamics, and cost modeling. They call for concerted action in dataset curation, architecture design, scaling, and modular/end-to-end evaluation to advance world-model-based learning in high-cost environments, highlighting the potential for safer, faster, and more reliable AI-driven progress across engineering and science.

Abstract

Large language model (LLM) agents trained using reinforcement learning has achieved superhuman performance in low-cost environments like games, mathematics, and coding. However, these successes have not translated to complex domains where the cost of interaction is high, such as the physical cost of running robots, the time cost of ML engineering, and the resource cost of scientific experiments. The true bottleneck for achieving the next level of agent performance for these complex and high-cost domains lies in the expense of executing actions to acquire reward signals. To address this gap, this paper argues that we should use world models as an intermediary between agents and the real world. We discuss how world models, viewed as models of dynamics, rewards, and task distributions, can overcome fundamental barriers of high-cost actions such as extreme off-policy learning and sample inefficiency in long-horizon tasks. Moreover, we demonstrate how world models can provide critical and rich learning signals to agents across a broad set of domains, including machine learning engineering, computer use, robotics, and AI for science. Lastly, we identify the challenges of building these world models and propose actionable items along dataset curation, architecture design, scaling, and evaluation of world models.

Figures (7)

• Figure 1: The Cost-Performance Gap. In low-cost environments with perfect simulators or low-cost execution, we have achieved superhuman agents such as AlphaGo silver2017mastering and gold-medal winning coding agent xu2025icpc. In high-cost environments such as robotics and ML engineering where execution of each action is constrained by the physical world or can take a long time, we are yet to achieve superhuman agent.
• Figure 2: World model as an intermediary between agents and the real world. In domains where the cost of interactions is high, agents should mostly interact with the world model for training and evaluation, and occasionally interact with the real world for real-world data collection and real-world validation. Most importantly, the real-world interaction data should be used not only to improve the agent but to improve the world model in its ability to predict the next state and reward.
• Figure 3: Components of a world model. A world model can consist of models of realistic task distribution $\hat{G}$, models of dynamics $\hat{T}$, models of reward $\hat{R}$, and models of cost. In code world models copet2025cwm, $\hat{G}, \hat{T}, \hat{R}$ represent realistic coding tasks, execution trace prediction, and performance/cost prediction. In molecular dynamics simulation razavian2012learning, $\hat{G}, \hat{T}, \hat{R}$ represent realistic initial experimental conditions, dynamics of atom movements, and property prediction such as formation energy. In robotics quevedo2506worldgym, $\hat{G}, \hat{T}, \hat{R}$ represent initial configuration of realistic scenes and language tasks, visual dynamics in response to actions, and prediction of success or failure.
• Figure 4: OOD tasks generated by a world model in WorldGym quevedo2506worldgym through image editing tools such as Nano Banana SharonEtAl2025GeminiImageEditing. New objects can be inserted to create novel initial observation $o_0$ and new language instructions can be curated to create a new task $g$. These OOD tasks can be used to evaluate, train, and debug policies (e.g., testing if they are better with colors or shapes).
• Figure 5: Real and generated computer UI for slide presentations from zhu2025paper2video. The generated application UI and the real application UI can look extremely similar, which provides an opportunity for a world model to faithfully simulate high-fidelity computer use environments, while ensuring proper sandboxing and safety guards.