Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WEBSERV: A Browser-Server Environment for Efficient Training of Reinforcement Learning-based Web Agents at Scale

Yuxuan Lu, Jing Huang, Hui Liu, Jiri Gesi, Yan Han, Shihan Fu, Tianqi Zheng, Dakuo Wang

TL;DR

We address a key bottleneck in reinforcement learning for web agents: the lack of scalable, reproducible full-stack environments that couple realistic browser interactions with isolated server-side state. WEBSERV combines a DOM-driven, compact observation space with a network-aware action executor and an Incus-based container manager to enable thousands of parallel RL rollouts. Empirical results on WebArena show state-of-the-art single-prompt performance and substantial resource-efficiency gains, including ~5x faster launches and ~240x storage reductions, while maintaining memory footprint and enabling 200+ concurrent containers. The framework also provides human-centered inspection tools (trajectory replay and real-time previews) to support safe transfer and debugging, improving reproducibility and practical deployment potential. Limitations include text-only observations and absence of direct visual modalities, suggesting future work to incorporate multimodal signals and richer layout cues.

Abstract

Training and evaluation of Reinforcement Learning (RL) web agents have gained increasing attention, yet a scalable and efficient environment that couples realistic and robust browser-side interaction with controllable server-side state at scale is still missing. Existing environments tend to have one or more of the following issues: they overwhelm policy models with excessive and noisy context; they perform actions non-deterministically without waiting for the UI or network to stabilize; or they cannot scale isolated client-server containers effectively for parallel RL rollouts. We propose WEBSERV, an environment that includes 1) a compact, site-agnostic browser environment that balances context and action complexity, and 2) a scalable RL environment via efficient launching and resetting web-servers to enable scalable RL training and evaluation. We evaluate WEBSERV on the shopping CMS and Gitlab tasks in WebArena, achieving state-of-the-art single-prompt success rates while cutting launch latency by ~5x and storage need by ~240x, with a comparable memory footprint, enabling 200+ concurrent containers on a single host.

WEBSERV: A Browser-Server Environment for Efficient Training of Reinforcement Learning-based Web Agents at Scale

TL;DR

We address a key bottleneck in reinforcement learning for web agents: the lack of scalable, reproducible full-stack environments that couple realistic browser interactions with isolated server-side state. WEBSERV combines a DOM-driven, compact observation space with a network-aware action executor and an Incus-based container manager to enable thousands of parallel RL rollouts. Empirical results on WebArena show state-of-the-art single-prompt performance and substantial resource-efficiency gains, including ~5x faster launches and ~240x storage reductions, while maintaining memory footprint and enabling 200+ concurrent containers. The framework also provides human-centered inspection tools (trajectory replay and real-time previews) to support safe transfer and debugging, improving reproducibility and practical deployment potential. Limitations include text-only observations and absence of direct visual modalities, suggesting future work to incorporate multimodal signals and richer layout cues.

Abstract

Training and evaluation of Reinforcement Learning (RL) web agents have gained increasing attention, yet a scalable and efficient environment that couples realistic and robust browser-side interaction with controllable server-side state at scale is still missing. Existing environments tend to have one or more of the following issues: they overwhelm policy models with excessive and noisy context; they perform actions non-deterministically without waiting for the UI or network to stabilize; or they cannot scale isolated client-server containers effectively for parallel RL rollouts. We propose WEBSERV, an environment that includes 1) a compact, site-agnostic browser environment that balances context and action complexity, and 2) a scalable RL environment via efficient launching and resetting web-servers to enable scalable RL training and evaluation. We evaluate WEBSERV on the shopping CMS and Gitlab tasks in WebArena, achieving state-of-the-art single-prompt success rates while cutting launch latency by ~5x and storage need by ~240x, with a comparable memory footprint, enabling 200+ concurrent containers on a single host.
Paper Structure (28 sections, 1 figure, 2 tables)

This paper contains 28 sections, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Web Environment Design
  4. Web Agents
  5. WebServ
  6. System Overview
  7. Environment
  8. Observation Space
  9. Filtering.
  10. Flattening and pruning.
  11. Interactivity detection.
  12. Semantic identifiers.
  13. Javascript state capture.
  14. Output schema.
  15. Action Space
  16. ...and 13 more sections

Figures (1)

  • Figure 1: System Architecture of WebServ. Each LLM Agent interact with an isolated pair of Browser Env and Web Server Container.