Current Agents Fail to Leverage World Model as Tool for Foresight

TL;DR

It is observed that some agents rarely invoke simulation, frequently misuse predicted rollouts, and often exhibit inconsistent or even degraded performance when simulation is available or enforced, highlighting the need for mechanisms that foster calibrated, strategic interaction with world models.

Abstract

Agents built on vision-language models increasingly face tasks that demand anticipating future states rather than relying on short-horizon reasoning. Generative world models offer a promising remedy: agents could use them as external simulators to foresee outcomes before acting. This paper empirically examines whether current agents can leverage such world models as tools to enhance their cognition. Across diverse agentic and visual question answering tasks, we observe that some agents rarely invoke simulation (fewer than 1%), frequently misuse predicted rollouts (approximately 15%), and often exhibit inconsistent or even degraded performance (up to 5%) when simulation is available or enforced. Attribution analysis further indicates that the primary bottleneck lies in the agents' capacity to decide when to simulate, how to interpret predicted outcomes, and how to integrate foresight into downstream reasoning. These findings underscore the need for mechanisms that foster calibrated, strategic interaction with world models, paving the way toward more reliable anticipatory cognition in future agent systems.

Figures (13)

• Figure 1: The world model as tool framework where the agent decides between real action taking and simulation.
• Figure 2: Percentage breakdown of the world model's impact. "WM Helps" is generally less frequent than "WM Hurts" for agent tasks, while VQA shows a more balanced distribution.
• Figure 3: Attributions for when world model use helps or hurts across Agent (top) and VQA (bottom) tasks.
• Figure 4: A Taxonomy of World Model Governance Successes: Correctly functioning governance follows a three-stage cognitive pipeline enabled by Strategic Input (I), Clear Interpretation (II), and Grounded Action (III). Success arises from calibrated queries (I), unambiguous verification (II), and stable integration of simulations into actionable plans (III).
• Figure 5: A Taxonomy of World Model Governance Failures: Pipeline breakdowns map to three disruptive pillars: Calibration Failures (I) causing unnecessary or missed simulation, Interpretation Ambiguity (II) corrupting signal-to-decision alignment, and Unstable Integration Policy (III) preventing foresight from becoming sustained progress. The dominant zones indicate the key bottleneck is governance stability, instead of foresight generation.