DynaWeb: Model-Based Reinforcement Learning of Web Agents

TL;DR

This work tackles the high cost and risk of training web agents through live web interaction by introducing DynaWeb, a model-based RL framework that learns a web world model to generate imaginative, multi-step rollouts. The policy is trained on a mixture of imagined trajectories and real expert data, using GSPO for stable on-policy optimization. The world model is explicitly trained to predict web dynamics via state-change descriptions on structured accessibility-tree representations, enabling realistic simulations and effective learning. Across WebArena and WebVoyager, DynaWeb yields consistent performance gains and provides insights into the appropriate dream length and the value of grounding learning with environment-specific world models.

Abstract

The development of autonomous web agents, powered by Large Language Models (LLMs) and reinforcement learning (RL), represents a significant step towards general-purpose AI assistants. However, training these agents is severely hampered by the challenges of interacting with the live internet, which is inefficient, costly, and fraught with risks. Model-based reinforcement learning (MBRL) offers a promising solution by learning a world model of the environment to enable simulated interaction. This paper introduces DynaWeb, a novel MBRL framework that trains web agents through interacting with a web world model trained to predict naturalistic web page representations given agent actions. This model serves as a synthetic web environment where an agent policy can dream by generating vast quantities of rollout action trajectories for efficient online reinforcement learning. Beyond free policy rollouts, DynaWeb incorporates real expert trajectories from training data, which are randomly interleaved with on-policy rollouts during training to improve stability and sample efficiency. Experiments conducted on the challenging WebArena and WebVoyager benchmarks demonstrate that DynaWeb consistently and significantly improves the performance of state-of-the-art open-source web agent models. Our findings establish the viability of training web agents through imagination, offering a scalable and efficient way to scale up online agentic RL.

Figures (6)

• Figure 1: Comparison between traditional web agent training via live web interaction and DynaWeb. By replacing risky and inefficient real-world interaction with a learned web world model, DynaWeb enables imagination-driven training using virtual pages and dreamed trajectories, optionally augmented with real expert data, resulting in safer and more efficient agent optimization.
• Figure 2: Overview of DynaWeb. DynaWeb trains web agents via imagination-driven, model-based reinforcement learning. A learned web world model serves as a synthetic environment, enabling the agent to generate multi-step imagined rollouts without interacting with the live web. These imagined trajectories are mixed with a small fraction of real expert trajectories to stabilize learning. The agent policy is optimized using sequence-level policy optimization, allowing efficient and robust credit assignment for long-horizon web tasks with sparse terminal rewards.
• Figure 3: Effect of world model training on downstream agent performance. Success rate (%) comparing a supervised task-specific world model (DynaWeb WM) and a frozen general-purpose LLM (GPT-oss-120b).
• Figure 4: System prompt used for training and evaluation of the WebArena agent.
• Figure 5: System prompt used to train the web world model for predicting next-step accessibility trees from actions and current observations.