Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Agentic Reward Modeling: Verifying GUI Agent via Online Proactive Interaction

Chaoqun Cui, Jing Huang, Shijing Wang, Liming Zheng, Qingchao Kong, Zhixiong Zeng

TL;DR

This work tackles the limitations of rule-based and passive LLM-based GUI evaluation by introducing Agentic Interactive Verification via the VAGEN framework, where a verifier agent actively probes environment states to verify task completion. VAGEN uses a memory-consolidation module, a tool-augmented verifier with progressive verification, and test-time scaling mechanisms to improve reliability and efficiency of reward modeling for GUI agents. The approach yields substantial accuracy gains on OSWorld-Verified and AndroidWorld, with read-only and reward-guided scaling providing strong performance and sample efficiency guarantees; its theoretical analysis derives a closed-form expression for final success rate under Best-of-$N$ sampling. Overall, VAGEN offers a robust, backbone-agnostic paradigm for GUI automation evaluation, enabling more reliable RL signals and scalable deployment while emphasizing safe, sandboxed operation.

Abstract

Reinforcement learning with verifiable rewards (RLVR) is pivotal for the continuous evolution of GUI agents, yet existing evaluation paradigms face significant limitations. Rule-based methods suffer from poor scalability and cannot handle open-ended tasks, while LLM-as-a-Judge approaches rely on passive visual observation, often failing to capture latent system states due to partial state observability. To address these challenges, we advocate for a paradigm shift from passive evaluation to Agentic Interactive Verification. We introduce VAGEN, a framework that employs a verifier agent equipped with interaction tools to autonomously plan verification strategies and proactively probe the environment for evidence of task completion. Leveraging the insight that GUI tasks are typically "easy to verify but hard to solve", VAGEN overcomes the bottlenecks of visual limitations. Experimental results on OSWorld-Verified and AndroidWorld benchmarks demonstrate that VAGEN significantly improves evaluation accuracy compared to LLM-as-a-Judge baselines and further enhances performance through test-time scaling strategies.

Agentic Reward Modeling: Verifying GUI Agent via Online Proactive Interaction

TL;DR

This work tackles the limitations of rule-based and passive LLM-based GUI evaluation by introducing Agentic Interactive Verification via the VAGEN framework, where a verifier agent actively probes environment states to verify task completion. VAGEN uses a memory-consolidation module, a tool-augmented verifier with progressive verification, and test-time scaling mechanisms to improve reliability and efficiency of reward modeling for GUI agents. The approach yields substantial accuracy gains on OSWorld-Verified and AndroidWorld, with read-only and reward-guided scaling providing strong performance and sample efficiency guarantees; its theoretical analysis derives a closed-form expression for final success rate under Best-of- sampling. Overall, VAGEN offers a robust, backbone-agnostic paradigm for GUI automation evaluation, enabling more reliable RL signals and scalable deployment while emphasizing safe, sandboxed operation.

Abstract

Reinforcement learning with verifiable rewards (RLVR) is pivotal for the continuous evolution of GUI agents, yet existing evaluation paradigms face significant limitations. Rule-based methods suffer from poor scalability and cannot handle open-ended tasks, while LLM-as-a-Judge approaches rely on passive visual observation, often failing to capture latent system states due to partial state observability. To address these challenges, we advocate for a paradigm shift from passive evaluation to Agentic Interactive Verification. We introduce VAGEN, a framework that employs a verifier agent equipped with interaction tools to autonomously plan verification strategies and proactively probe the environment for evidence of task completion. Leveraging the insight that GUI tasks are typically "easy to verify but hard to solve", VAGEN overcomes the bottlenecks of visual limitations. Experimental results on OSWorld-Verified and AndroidWorld benchmarks demonstrate that VAGEN significantly improves evaluation accuracy compared to LLM-as-a-Judge baselines and further enhances performance through test-time scaling strategies.
Paper Structure (39 sections, 1 theorem, 15 equations, 12 figures, 6 tables)

This paper contains 39 sections, 1 theorem, 15 equations, 12 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Computer Use Agent for GUI Automation Tasks
  4. Verifiable Rewards for GUI Agents
  5. Method
  6. Problem Formulation
  7. Overall Framework
  8. Trajectory Memory Consolidation
  9. Verifier Action Space
  10. Progressive Verification Mechanism
  11. Test-Time Scaling
  12. Read-Only Scaling of Verifier Agent
  13. Reward-Guided Scaling of Actor Agent
  14. Experiments
  15. Experimental Settings
  16. ...and 24 more sections

Key Result

Theorem 3.1

Let $p\in [0, 1]$ be the inherent success rate of an actor agent and $a\in [0, 1]$ be the accuracy of a reward model. Under $N$ attempts guided by the reward model as a supervisor, the final success rate $P_{\text{final}}(N)$ of the actor agent is given by: where $\alpha =pa+(1-p)(1-a)$ and $\beta=1 - \alpha$ are the marginal probabilities of positive and negative judgements.

Figures (12)

  • Figure 1: The evolution of GUI agent verification methods. Unlike (a) Rule-based and (b) LLM-as-a-Judge approaches which suffer from scalability issues or partial observability, (c) Agentic Interactive Verification (VAGEN) introduces a verifier agent capable of proactively probing the environment to overcome visual limitations.
  • Figure 2: Workflow of the Progressive Verification Mechanism for the task "Help me buy the book Reinforcement Learning by Richard".
  • Figure 3: Illustration of the Reward-Guided Scaling strategy.
  • Figure 4: Read-only scaling for verifier agent (Claude-Sonnet-4.5).
  • Figure 5: Results of reward-guided test-time scaling for the actor agent using Best-of-N rejection sampling.
  • ...and 7 more figures

Theorems & Definitions (1)

  • Theorem 3.1