Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Emergence WebVoyager: Toward Consistent and Transparent Evaluation of (Web) Agents in The Wild

Deepak Akkil, Mowafak Allaham, Amal Raj, Tamer Abuelsaad, Ravi Kokku

Abstract

Reliable evaluation of AI agents operating in complex, real-world environments requires methodologies that are robust, transparent, and contextually aligned with the tasks agents are intended to perform. This study identifies persistent shortcomings in existing AI agent evaluation practices that are particularly acute in web agent evaluation, as exemplified by our audit of WebVoyager, including task-framing ambiguity and operational variability that hinder meaningful and reproducible performance comparisons. To address these challenges, we introduce Emergence WebVoyager, an enhanced version of the WebVoyager benchmark that standardizes evaluation methodology through clear guidelines for task instantiation, failure handling, annotation, and reporting. Emergence WebVoyager achieves an inter-annotator agreement of 95.9\%, indicating improved clarity and reliability in both task formulation and evaluation. Applying this framework to evaluate OpenAI Operator reveals substantial performance variation across domains and task types, with an overall success rate of 68.6\%, substantially lower than the 87\% previously reported by OpenAI, demonstrating the utility of our approach for more rigorous and comparable web agent evaluation.

Emergence WebVoyager: Toward Consistent and Transparent Evaluation of (Web) Agents in The Wild

Abstract

Reliable evaluation of AI agents operating in complex, real-world environments requires methodologies that are robust, transparent, and contextually aligned with the tasks agents are intended to perform. This study identifies persistent shortcomings in existing AI agent evaluation practices that are particularly acute in web agent evaluation, as exemplified by our audit of WebVoyager, including task-framing ambiguity and operational variability that hinder meaningful and reproducible performance comparisons. To address these challenges, we introduce Emergence WebVoyager, an enhanced version of the WebVoyager benchmark that standardizes evaluation methodology through clear guidelines for task instantiation, failure handling, annotation, and reporting. Emergence WebVoyager achieves an inter-annotator agreement of 95.9\%, indicating improved clarity and reliability in both task formulation and evaluation. Applying this framework to evaluate OpenAI Operator reveals substantial performance variation across domains and task types, with an overall success rate of 68.6\%, substantially lower than the 87\% previously reported by OpenAI, demonstrating the utility of our approach for more rigorous and comparable web agent evaluation.

Paper Structure

This paper contains 20 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Limitations of WebVoyager for Web-Agent Evaluation
  4. Task framing ambiguity
  5. Website of Execution is Not Enforced
  6. Ambiguous Task Definitions
  7. Reliance on Static Dates
  8. Temporal Sensitivity of Tasks
  9. Prevalence of Relatively Easy Tasks in the Original Benchmark
  10. Operational Variability in Execution
  11. Location of Benchmark Execution
  12. Handling of External Constraints such as CAPTCHAs
  13. Differences in Evaluation Protocols
  14. Interpretation of Task Success Can Be Subjective
  15. Task Update and Exclusion Criteria
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Evaluation of OpenAI Operator on Emergence WebVoyager. Figure 1(a) shows the percentage of tasks successfully completed by Operator on each website. Figure 1(b) reports the average task completion times per website for successful and failed tasks
  • Figure 2: Annotation interface of the tool we developed and used for evaluating Operator performance in Emergence WebVoyager.