Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Web-Shepherd: Advancing PRMs for Reinforcing Web Agents

Hyungjoo Chae, Sunghwan Kim, Junhee Cho, Seungone Kim, Seungjun Moon, Gyeom Hwangbo, Dongha Lim, Minjin Kim, Yeonjun Hwang, Minju Gwak, Dongwook Choi, Minseok Kang, Gwanhoon Im, ByeongUng Cho, Hyojun Kim, Jun Hee Han, Taeyoon Kwon, Minju Kim, Beong-woo Kwak, Dongjin Kang, Jinyoung Yeo

TL;DR

Web navigation poses long-horizon decision challenges for agents and existing reward signals are either costly (LLM-based evaluators) or sparse. The authors introduce Web-Shepherd, a first process reward model (PRM) for web trajectories, paired with the WebPRM Collection and WebRewardBench to enable training and evaluation of step-level rewards. Empirical results show Web-Shepherd achieves higher reward-prediction accuracy than prompting GPT-4o and substantially improves reward-guided trajectory search with an order-of-magnitude cost reduction. The work provides open datasets, code, and a systematic analysis of checklist quality, training objectives, and data requirements, offering a practical pathway to more reliable web agent systems.

Abstract

Web navigation is a unique domain that can automate many repetitive real-life tasks and is challenging as it requires long-horizon sequential decision making beyond typical multimodal large language model (MLLM) tasks. Yet, specialized reward models for web navigation that can be utilized during both training and test-time have been absent until now. Despite the importance of speed and cost-effectiveness, prior works have utilized MLLMs as reward models, which poses significant constraints for real-world deployment. To address this, in this work, we propose the first process reward model (PRM) called Web-Shepherd which could assess web navigation trajectories in a step-level. To achieve this, we first construct the WebPRM Collection, a large-scale dataset with 40K step-level preference pairs and annotated checklists spanning diverse domains and difficulty levels. Next, we also introduce the WebRewardBench, the first meta-evaluation benchmark for evaluating PRMs. In our experiments, we observe that our Web-Shepherd achieves about 30 points better accuracy compared to using GPT-4o on WebRewardBench. Furthermore, when testing on WebArena-lite by using GPT-4o-mini as the policy and Web-Shepherd as the verifier, we achieve 10.9 points better performance, in 10 less cost compared to using GPT-4o-mini as the verifier. Our model, dataset, and code are publicly available at LINK.

Web-Shepherd: Advancing PRMs for Reinforcing Web Agents

TL;DR

Web navigation poses long-horizon decision challenges for agents and existing reward signals are either costly (LLM-based evaluators) or sparse. The authors introduce Web-Shepherd, a first process reward model (PRM) for web trajectories, paired with the WebPRM Collection and WebRewardBench to enable training and evaluation of step-level rewards. Empirical results show Web-Shepherd achieves higher reward-prediction accuracy than prompting GPT-4o and substantially improves reward-guided trajectory search with an order-of-magnitude cost reduction. The work provides open datasets, code, and a systematic analysis of checklist quality, training objectives, and data requirements, offering a practical pathway to more reliable web agent systems.

Abstract

Web navigation is a unique domain that can automate many repetitive real-life tasks and is challenging as it requires long-horizon sequential decision making beyond typical multimodal large language model (MLLM) tasks. Yet, specialized reward models for web navigation that can be utilized during both training and test-time have been absent until now. Despite the importance of speed and cost-effectiveness, prior works have utilized MLLMs as reward models, which poses significant constraints for real-world deployment. To address this, in this work, we propose the first process reward model (PRM) called Web-Shepherd which could assess web navigation trajectories in a step-level. To achieve this, we first construct the WebPRM Collection, a large-scale dataset with 40K step-level preference pairs and annotated checklists spanning diverse domains and difficulty levels. Next, we also introduce the WebRewardBench, the first meta-evaluation benchmark for evaluating PRMs. In our experiments, we observe that our Web-Shepherd achieves about 30 points better accuracy compared to using GPT-4o on WebRewardBench. Furthermore, when testing on WebArena-lite by using GPT-4o-mini as the policy and Web-Shepherd as the verifier, we achieve 10.9 points better performance, in 10 less cost compared to using GPT-4o-mini as the verifier. Our model, dataset, and code are publicly available at LINK.

Paper Structure

This paper contains 89 sections, 2 equations, 28 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. MLLM-based web agents.
  4. Inference-time scaling for web agents.
  5. Rewards for web navigation.
  6. Preliminaries
  7. WebPRM Collection
  8. Collecting User Instruction and Expert Trajectory
  9. Annotating Checklist and Rejected Action
  10. Checklist.
  11. Rejected actions.
  12. Dataset Statistics
  13. Web-Shepherd
  14. Step 1: Checklist Generation
  15. Step 2: Reward Modeling with Checklist
  16. ...and 74 more sections

Figures (28)

  • Figure 1: Performance and cost-efficiency of Web-Shepherd (3B). Web-Shepherd achieves the state-of-the-art performance while requiring significantly lower cost compared to existing baselines.
  • Figure 2: Example of web navigation under a POMDP.
  • Figure 3: Overview of the dataset collection process of WebPRM Collection (top) and an example instance of our dataset (bottom).
  • Figure 4: Statistics of WebPRM Collection.
  • Figure 5: Overview of Web-Shepherd (left) and its diverse use cases (right).
  • ...and 23 more figures