WebGym: Scaling Training Environments for Visual Web Agents with Realistic Tasks

TL;DR

WebGym provides the largest open-source platform for training visual web agents with nearly 300k real-web tasks and a high-throughput asynchronous rollout system. Using a simple REINFORCE-like RL approach and rubric-guided rewards, a base vision-language model achieves state-of-the-art generalization on unseen websites, outperforming proprietary models. The work demonstrates the importance of domain breadth, task difficulty depth, and training horizon in scaling agent performance, and offers practical guidelines for robust, scalable training of visual web agents. Its modular rollout design and memory-enabled prompting provide a blueprint for future improvements in rubric-based evaluation and reinforcement learning on live websites.

Abstract

We present WebGym, the largest-to-date open-source environment for training realistic visual web agents. Real websites are non-stationary and diverse, making artificial or small-scale task sets insufficient for robust policy learning. WebGym contains nearly 300,000 tasks with rubric-based evaluations across diverse, real-world websites and difficulty levels. We train agents with a simple reinforcement learning (RL) recipe, which trains on the agent's own interaction traces (rollouts), using task rewards as feedback to guide learning. To enable scaling RL, we speed up sampling of trajectories in WebGym by developing a high-throughput asynchronous rollout system, designed specifically for web agents. Our system achieves a 4-5x rollout speedup compared to naive implementations. Second, we scale the task set breadth, depth, and size, which results in continued performance improvement. Fine-tuning a strong base vision-language model, Qwen-3-VL-8B-Instruct, on WebGym results in an improvement in success rate on an out-of-distribution test set from 26.2% to 42.9%, significantly outperforming agents based on proprietary models such as GPT-4o and GPT-5-Thinking that achieve 27.1% and 29.8%, respectively. This improvement is substantial because our test set consists only of tasks on websites never seen during training, unlike many other prior works on training visual web agents.

Figures (12)

• Figure 1: Task sourcing for WebGym. We aggregate web agent task sets from 10 widely-used benchmarks and environments to seed our procedural construction. We report original statistics (#difficulties, #websites, #tasks) for each seed set used to construct the WebGym task set. Some task sets are marked with only 1 website because the original task sets do not provide task-specific website.
• Figure 2: Task decomposition system.WebGym decomposes tasks by generating valid combinations of fact groups from the original task's rubric. Decomposition requires $\geq$2 groups with at least one "large" group ($\geq$3 facts). Each valid combination (excluding the full set) produces a new task with lower difficulty while maintaining consistency with the original objectives.
• Figure 3: Statistics of the WebGymtraining task set.Left: website distribution as a function of the sorted index (from more tasks to less) of website. Right: difficulty distribution of the train and test task sets. The transparent bars over the original bar mean decomposed tasks, and the slashed bars means test set.]
• Figure 4: Analysis of the WebGym task set.Left: distribution of tasks across domains according to the Mind2Web-2 taxonomy mind2web-2. Right: Distribution of trajectory lengths by task difficulty for answered trajectories over multiple iterations. For trajectories working on medium- and hard-difficulty tasks that exceeds 30 tasks during evaluation, we filter in only trajectories under 30 tasks to make the comparison fair. The violin plots show that the probability density of trajectory lengths, with the KDE mode (black line, KDE mode is the peak of the smoothed probability density curve, where the violin is widest) indicating the most likely length and mean (red) indicating the average.
• Figure 5: Agreement between automated evaluators and human judgment. Rubric-based evaluation (with explicit criteria) consistently improves agreement over task-only evaluation, yielding higher accuracy and precision across LLM-based evaluators. Among the evaluators, GPT-4o shows the largest shift after adding the rubric: precision increases the most while recall drops, indicating that the rubric makes GPT-4o apply stricter, more conservative pass criteria.