Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation

TL;DR

This work identifies a critical gap in LLM-based web agents: the absence of world models that anticipate the outcomes of actions. It introduces World-Model-Augmented (WMA) web agents, which learn transition-focused environment dynamics and use a frozen policy model to simulate future observations and estimate rewards before acting. By training a dedicated world model with transition-focused observation abstractions and performing inference-time policy optimization, the approach achieves notable gains in WebArena and surpasses prior SOTA on Mind2Web, while substantially reducing exploration cost and time compared to tree-search baselines. The findings demonstrate the practical value of incorporating world models into web navigation and offer a foundation for future multimodal extensions and self-refinement strategies.

Abstract

Large language models (LLMs) have recently gained much attention in building autonomous agents. However, the performance of current LLM-based web agents in long-horizon tasks is far from optimal, often yielding errors such as repeatedly buying a non-refundable flight ticket. By contrast, humans can avoid such an irreversible mistake, as we have an awareness of the potential outcomes (e.g., losing money) of our actions, also known as the "world model". Motivated by this, our study first starts with preliminary analyses, confirming the absence of world models in current LLMs (e.g., GPT-4o, Claude-3.5-Sonnet, etc.). Then, we present a World-model-augmented (WMA) web agent, which simulates the outcomes of its actions for better decision-making. To overcome the challenges in training LLMs as world models predicting next observations, such as repeated elements across observations and long HTML inputs, we propose a transition-focused observation abstraction, where the prediction objectives are free-form natural language descriptions exclusively highlighting important state differences between time steps. Experiments on WebArena and Mind2Web show that our world models improve agents' policy selection without training and demonstrate our agents' cost- and time-efficiency compared to recent tree-search-based agents.

Figures (23)

• Figure 1: LLMs' performance in next state prediction.
• Figure 2: LLMs' performance in action selection (w/ and w/o next states).
• Figure 3: Framework overview. We first collect training data for world models (top). After training, we perform policy optimization by selecting the action leading to an optimal next state (bottom).
• Figure 4: Sequence length distribution of different observation representations.
• Figure 5: The overview of transition-focused observation abstraction.