CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using Agent

TL;DR

CUARewardBench introduces the first benchmark for evaluating both outcome and process reward models on computer-using agents, using a diverse OSWorld-based dataset across 10 software categories and 7 agent architectures. It analyzes 7 vision-language models with 3 prompt templates, uncovering limitations in current reward models—most notably visual reasoning gaps and knowledge deficiencies—and shows general VLMs often outperform CUA-specialized models. The authors propose Unanimous Prompt Ensemble (UPE), a strict unanimous voting scheme with prompt diversification, achieving notable gains in precision and negative predictive value for both ORM and PRM. Limitations include a relatively small scale and the absence of RL validation for UPE, but the work provides a rigorous testbed and a practical method to enhance reward reliability for large-scale CUA training and deployment.

Abstract

Computer-using agents (CUAs) enable task completion through natural interaction with operating systems and software interfaces. While script-based verifiers are widely adopted for evaluation, they suffer from limited scalability and inability to provide step-wise assessment. Reward models offer promising alternatives, but their effectiveness on CUA evaluation remains largely underexplored. To address this gap, we present CUARewardBench, comprising four key contributions: (1) First-ever Comprehensive CUA Reward Benchmark: We introduce the first benchmark for evaluating both outcome reward models (ORM) and process reward models (PRM) on CUA tasks, enabling systematic assessment across trajectory-level and step-level evaluation. (2) Diverse, Practical and Reliable Dataset: CUARewardBench encompasses trajectories from 10 software categories and 7 agent architectures with varying performance levels (25.9%-50.8% success rates). All trajectories are expertly annotated through carefully designed protocols, with rigorous quality control to ensure reliability and practical applicability. (3) Comprehensive Analysis and Insights: Through extensive experiments across 7 vision-language models and 3 prompt templates, we reveal critical limitations of current CUA RMs, including insufficient visual reasoning capabilities, knowledge deficiencies, and the superiority of general VLMs over specialized CUA models for reward evaluation. (4) Unanimous Prompt Ensemble (UPE): Based on the insights from our comprehensive analysis, we propose UPE, a novel ensemble method that significantly enhances reward model reliability through strict unanimous voting and strategic prompt-template configurations. UPE achieves 89.8% precision and 93.3% NPV for ORM, and 81.7% precision and 85.1% NPV for PRM, substantially outperforming single VLMs and traditional ensemble approaches.

Figures (6)

• Figure 1: UPE achieves superior reward model reliability. Performance comparison on ORM and PRM tasks shows that our proposed UPE (red star) simultaneously achieves high precision and NPV, significantly outperforming single VLMs with different prompts and traditional ensemble methods (majority-voting). The upper-right positioning demonstrates UPE's effectiveness in balancing positive and negative prediction accuracy. Details of UPE are discussed in Section \ref{['sec:ensemble_methods']}.
• Figure 2: CUARewardBench dataset characteristics and selected experimental findings: (a) Task distribution of trajectory annotations across 10 software categories (Section \ref{['sec:annotation']}), (b) policy model diversity in our benchmark (Section \ref{['sec:traj_collection']}), and (c) key error modes identified in our evaluation experiments (Section \ref{['sec:error_analysis']}).
• Figure 3: ORM prompt of ZeroGUI zerogui. The prompt instructs the vision-language model to analyze computer usage task completion through detailed screenshot examination and structured response formatting.
• Figure 4: SE-WSM seagent prompt template both for ORM and PRM evaluation. The prompt instructs the vision-language model to conduct multi-dimensional assessment including trajectory correctness, redundant steps identification, first error step detection, and correct action suggestions for both ORM and PRM evaluation.
• Figure 5: original PRM Prompts of OpenCUA reflector opencua. The prompt provides comprehensive step-level assessment including thought process analysis, action-code alignment verification, and structured JSON output format for systematic evaluation of agent decision-making processes.