Scaling Web Agent Training through Automatic Data Generation and Fine-grained Evaluation

TL;DR

This work tackles the scalability bottleneck in training web agents by combining automatic trajectory data generation with a fine-grained constraint-based evaluation framework. The authors introduce BookingArena, a 120-task real-world booking benchmark, and demonstrate that a 24B parameter student model trained through LoRA distillation on partially successful trajectories can outperform open-source and match commercial systems on complex, long-horizon tasks. A key contribution is the CSR-based evaluation that enables leveraging partial progress and applying hindsight relabeling to enrich training data without requiring manual labeling. The results show that constraint-aware data curation and distillation yield efficient, effective web agents, with broader implications for scalable training of agents in real-world web environments.

Abstract

We present a scalable pipeline for automatically generating high-quality training data for web agents. In particular, a major challenge in identifying high-quality training instances is trajectory evaluation - quantifying how much progress was made towards task completion. We introduce a novel constraint-based evaluation framework that provides fine-grained assessment of progress towards task completion. This enables us to leverage partially successful trajectories, which significantly expands the amount of usable training data. We evaluate our method on a new benchmark we propose called BookingArena, which consists of complex booking tasks across 20 popular websites, and demonstrate that our distilled student model outperforms open-source approaches and matches or exceeds commercial systems, while being a significantly smaller model. Our work addresses the challenge of efficiently creating diverse, realistic web interaction datasets and provides a systematic evaluation methodology for complex structured web tasks.

Figures (9)

• Figure 1: Overview. We introduce a scalable pipeline to automatically generate and evaluate web agent trajectories for training competent small language models (\ref{['sec:pipeline']}). Given trajectories generated by a few-shot prompted teacher agent, our novel constraint-based fine-grained evaluation framework (\ref{['sec:constraint_framework']}) extracts high-quality training instances for distillation. See text for details.
• Figure 2: Prefix Extraction with Constraint Evaluation: For a given trajectory, we compute the constraint satisfaction rate (CSR) for each time-step and extract the smallest prefix that reaches maximum CSR. In this example, only the first two actions are retained and the third action is discarded as it results in a decrease of CSR (Agent should have clicked on the 'search' button instead).
• Figure 3: Task generation prompt template used to generate diverse tasks for each website. The template includes specific criteria for task generation and ensures consistency across different difficulty levels while maintaining website-specific rules.
• Figure 4: Prompt used for few-shot agent.
• Figure 5: Prompt used for constraint evaluation with screenshot observations and VLM judge.