Dyna-Think: Synergizing Reasoning, Acting, and World Model Simulation in AI Agents

TL;DR

Dyna-Think introduces a two-stage framework that tightly couples reasoning, action, and internal world-model simulation to improve long-horizon AI agents. By first distilling world-model-centric thinking (DIT) and then performing online Dyna-style training (DDT) that jointly updates policy and world-model components, the approach achieves competitive best-of-n performance with far fewer tokens than a much larger model. Critique-based world-model training emerges as a particularly effective signal, and stronger world-model accuracy correlates with higher task success, supporting the central claim that integrated world-model simulation enhances planning and acting. Across OSWorld and WindowsAgentArena benchmarks, Dyna-Think demonstrates improved in-domain and out-of-domain performance and token efficiency, suggesting a promising direction for scalable, reasoning-enabled AI agents.

Abstract

Recent progress in reasoning with large language models (LLMs), such as DeepSeek-R1, demonstrates impressive capabilities in domains like mathematics and coding, by exhibiting complex cognitive behaviors such as verification, goal decomposition, and self-reflection. However, it is unclear what behavior is effective and what behavior is missing for long-horizon AI agents tasks. In this work, we propose Dyna-Think, a thinking framework that integrates planning with an internal world model with reasoning and acting to enhance AI agent performance. To enable Dyna-Think, we propose Dyna-Think Imitation Learning (DIT) and Dyna-Think Dyna Training (DDT). To initialize a policy with Dyna-Think, DIT reconstructs the thinking process of R1 to focus on performing world model simulation relevant to the proposed (and planned) action, and trains the policy using this reconstructed data. To enhance Dyna-Think, DDT uses a two-stage training process to first improve the agent's world modeling ability via objectives such as state prediction or critique generation, and then improve the agent's action via policy training. We evaluate our methods on OSWorld and WindowsAgentArena, and demonstrate that Dyna-Think improves the agent's in-domain and out-of-domain performance, achieving similar best-of-n performance compared to R1 while generating 2x less tokens on average. Our extensive empirical studies reveal that 1) using critique generation for world model training is effective to improve policy performance; and 2) AI agents with better performance correlate with better world modeling abilities. We believe our results suggest a promising research direction to integrate world model simulation into AI agents to enhance their reasoning, planning, and acting capabilities.

Figures (9)

• Figure 1: Dyna-Think focus on integrating world model simulation (shown in green) with reasoning and acting. Behaviors that is not necessary to cause/has unclear contribution to the final action is shown in orange. After training, we find Dyna-Think achieves similar BoN performance compared to R1, while generating 2x less tokens on average and being only 32B in size.
• Figure 2: Our Dyna-Think framework synergizes planning with world model simulation in an agent's reasoning process, and performs both world model training and policy training with $\pi_{\mathcal{W}}(\theta)$.
• Figure 3: Three different forms of world model data experimented in DDT.
• Figure 4: Scaling world model learning with synthetic tasks generated by GPT-4o. We use DDT($\hat{\mathcal{T}}_{\textrm{critic}}$) and train from our best model in \ref{['tbl:enable_v1']}. After world model training, we perform one-round of policy training using the same set of policy learning data.
• Figure 5: Iterative policy learning with and without adding evaluation configuration as hints. The added evaluation dictionary (in blue) is part of the task configuration provided by OSWorld.