WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Model

TL;DR

WebEvolver addresses stagnation in autonomous web-agent self-improvement by co-training a world-model LLM that predicts next observations and generates synthetic trajectories. The world model acts both as a virtual web server for training data and as an imagination engine for inference-time look-ahead, enabling deeper multi-step planning without heavy real-world interaction. Experiments across Mind2Web-Live, WebVoyager, GAIA-web, and SimpleQA show around a 10% performance improvement over baselines, with detailed analysis of world-model quality and generalization. The work underscores the importance of integrating dynamic world models into agent frameworks to sustain adaptability, without distillation from stronger, closed-source models.

Abstract

Agent self-improvement, where the backbone Large Language Model (LLM) of the agent are trained on trajectories sampled autonomously based on their own policies, has emerged as a promising approach for enhancing performance. Recent advancements, particularly in web environments, face a critical limitation: their performance will reach a stagnation point during autonomous learning cycles, hindering further improvement. We argue that this stems from limited exploration of the web environment and insufficient exploitation of pre-trained web knowledge in LLMs. To improve the performance of self-improvement, we propose a novel framework that introduces a co-evolving World Model LLM. This world model predicts the next observation based on the current observation and action within the web environment. Leveraging LLMs' pretrained knowledge of abundant web content, the World Model serves dual roles: (1) as a virtual web server generating self-instructed training data to continuously refine the agent's policy, and (2) as an imagination engine during inference, enabling look-ahead simulation to guide action selection for the agent LLM. Experiments in real-world web environments (Mind2Web-Live, WebVoyager, and GAIA-web) show a 10% performance gain over existing self-evolving agents, demonstrating the efficacy and generalizability of our approach, without using any distillation from more powerful close-sourced models. Our work establishes the necessity of integrating world models into autonomous agent frameworks to unlock sustained adaptability. Code is available at https://github.com/Tencent/SelfEvolvingAgent

Figures (4)

• Figure 1: Overview of WebEvolver – A Self-Improving Framework with World-Model Look-Ahead. Our framework co-trains a world model with the agent to predict next-step observations based on current states and actions. The world model serves as a virtual web engine, which generates synthetic trajectories for policy training and enables look-ahead planning to select optimal actions during inference.
• Figure 2: An illustration of the World Model trajectory synthesizing process and World Model Look-ahead for inference-time action selection.
• Figure 3: Visual illustration of overall success rate evolving on WebVoyager and Mind2Web-Live.
• Figure 4: An example of world model-synthesized trajectory.