WebArbiter: A Principle-Guided Reasoning Process Reward Model for Web Agents

TL;DR

WebArbiter tackles the challenge of sparse, long-horizon feedback in web navigation by introducing a reasoning-first, principle-inducing Process Reward Model. It reframes reward modeling as structured text generation that outputs principled justifications and a single verdict, trained in two stages: reasoning distillation and reinforcement learning via GRPO. The authors release WebPRMBench, a diverse benchmark with 1,150 preference instances across four web environments, and demonstrate state-of-the-art performance on BoN and Pairwise accuracy, plus substantial gains in reward-guided trajectory search on WebArena-Lite. The approach improves robustness to layout and semantic changes, provides interpretable credit assignment, and scales better to real-world web tasks than prior WebPRMs or LLM judges.

Abstract

Web agents hold great potential for automating complex computer tasks, yet their interactions involve long-horizon, sequential decision-making with irreversible actions. In such settings, outcome-based supervision is sparse and delayed, often rewarding incorrect trajectories and failing to support inference-time scaling. This motivates the use of Process Reward Models (WebPRMs) for web navigation, but existing approaches remain limited: scalar WebPRMs collapse progress into coarse, weakly grounded signals, while checklist-based WebPRMs rely on brittle template matching that fails under layout or semantic changes and often mislabels superficially correct actions as successful, providing little insight or interpretability. To address these challenges, we introduce WebArbiter, a reasoning-first, principle-inducing WebPRM that formulates reward modeling as text generation, producing structured justifications that conclude with a preference verdict and identify the action most conducive to task completion under the current context. Training follows a two-stage pipeline: reasoning distillation equips the model with coherent principle-guided reasoning, and reinforcement learning corrects teacher biases by directly aligning verdicts with correctness, enabling stronger generalization. To support systematic evaluation, we release WebPRMBench, a comprehensive benchmark spanning four diverse web environments with rich tasks and high-quality preference annotations. On WebPRMBench, WebArbiter-7B outperforms the strongest baseline, GPT-5, by 9.1 points. In reward-guided trajectory search on WebArena-Lite, it surpasses the best prior WebPRM by up to 7.2 points, underscoring its robustness and practical value in real-world complex web tasks.

Figures (7)

• Figure 1: Performance comparison on WebPRMBench. Left:Average Best-of-N Acc vs. model size, showing superior efficiency despite smaller scale. Right: Domain-wise Avg BoN Acc, where WebArbiter achieves the best results across all environments, confirming robustness and scalability.
• Figure 2: Overview of WebArbiter. Given an instruction $\mathcal{I}$, current observation $o_p$, and history $(a_{<p},c_{<p})$, the model compares candidate actions $(a_p^1,c_p^1)$ and $(a_p^2,c_p^2)$. In Stage 1, principle-guided reasoning traces are distilled from a stronger teacher LLM. In Stage 2, WebArbiter is trained with RL using verifiable rewards $R \in \{-1,+1\}$, producing structured justifications and a final verdict. During inference, the model induces principles (e.g., clarity, correctness, progress) from $(\mathcal{I},o_p,a_{<p},c_{<p},(a_p^1,c_p^1),(a_p^2,c_p^2))$, applies them to candidate actions, and outputs an auditable judgment identifying the action that best advances task completion.
• Figure 3: Action-type distribution in WebPRMBench.
• Figure 4: Correlation between BoN and Pairwise Acc across web benchmarks. Each scatter point corresponds to a PRM. We report the correlation coefficient $r$ for each environment. While the two metrics are strongly correlated across all environments, BoN exhibits higher variance and provides finer-grained discrimination among models, particularly in complex web environments.
• Figure 5: Inference-time scaling of WebArbiter. Left:Pairwise and Right:BoN Acc as the number of sampled reward evaluations $K$ increases.