BrowserArena: Evaluating LLM Agents on Real-World Web Navigation Tasks

TL;DR

This work presents BrowserArena, a live open-web evaluation platform that pairs LLM web agents on user-submitted tasks to produce a human-in-the-loop, pairwise leaderboard. It demonstrates that VLM-based judges do not reliably reflect human preferences and introduces step-level annotations to identify recurring failure modes, enabling the construction of targeted datasets for captcha solving, banner closure, and direct navigation. The findings reveal both the behavioral diversity and brittleness of current web agents, and the methodology provides a scalable approach to understanding and diagnosing web-navigation failures at scale. Overall, BrowserArena advances open-web agent benchmarking by combining live tasks, human feedback, and failure-mode analysis to guide future improvements in web-agent capabilities.

Abstract

LLM web agents now browse and take actions on the open web, yet current agent evaluations are constrained to sandboxed environments or artificial tasks. We introduce BrowserArena, a live open-web agent evaluation platform that collects user-submitted tasks, runs Arena-style head-to-head comparisons, and uses step-level human feedback to surface failure modes. Collecting and analyzing step-level annotations on the agent traces, we identify three consistent failure modes: captcha resolution, pop-up banner removal, and direct navigation to URLs. By constructing targeted datasets to further study these tasks, we discover variations in how different language models navigate these failure modes. We find, for example, that o4-mini deploys a wider variety of strategies to circumvent captcha resolution than other models and DeepSeek-R1 consistently misleads users about pop-up banner closure. Our findings surface both the diversity and brittleness of current web agents. More broadly, our benchmarking methodology provides an approach to evaluating and understanding web agent failure modes at scale.

Figures (4)

• Figure 1: An overview of the study procedure showing how users interact with BrowserArena. We include examples of user submitted tasks in Appendix \ref{['app:example_user_tasks']}.
• Figure 2: We compute the average win rate, battle counts, bootstrapped ELO ratings, and pairwise win fractions from 109 user-submitted tasks and evaluations on Prolific. For the ELO-based leaderboard, we simply sort the models from highest to lowest bootstrapped ELO rating in Figure \ref{['fig:leaderboard_results']}(c).
• Figure 3: Pairwise agreements between the baseline labels, the new annotators, and two vision-languages models (GPT-4o and o4-mini; we take the majority @5).
• Figure 4: Failure mode identification with Docent meng2025docent.