DR-Arena: an Automated Evaluation Framework for Deep Research Agents

TL;DR

DR-Arena tackles the challenge of evaluating autonomous Deep Research agents in open, dynamic environments by introducing a fully automated evaluation framework. It pivots on a Automated Examiner that builds Dynamic Information Trees from live web data, generates Deep & Wide tasks, and uses an Adaptive Evolvement Loop to escalates Depth and Width based on real-time performance, thereby stress-testing core DR capabilities. In extensive experiments with six state-of-the-art DR agents, DR-Arena achieves a Spearman correlation of $0.94$ with human LMSYS leaderboards and demonstrates superior alignment to human preferences without manual adjudication. The framework also includes ablation and human validation studies, showing that the full tree, rubric-based judgments, and iterative rounds are essential for reliable rankings, making DR-Arena a scalable, robust proxy for expensive human evaluation in future DR systems.

Abstract

As Large Language Models (LLMs) increasingly operate as Deep Research (DR) Agents capable of autonomous investigation and information synthesis, reliable evaluation of their task performance has become a critical bottleneck. Current benchmarks predominantly rely on static datasets, which suffer from several limitations: limited task generality, temporal misalignment, and data contamination. To address these, we introduce DR-Arena, a fully automated evaluation framework that pushes DR agents to their capability limits through dynamic investigation. DR-Arena constructs real-time Information Trees from fresh web trends to ensure the evaluation rubric is synchronized with the live world state, and employs an automated Examiner to generate structured tasks testing two orthogonal capabilities: Deep reasoning and Wide coverage. DR-Arena further adopts Adaptive Evolvement Loop, a state-machine controller that dynamically escalates task complexity based on real-time performance, demanding deeper deduction or wider aggregation until a decisive capability boundary emerges. Experiments with six advanced DR agents demonstrate that DR-Arena achieves a Spearman correlation of 0.94 with the LMSYS Search Arena leaderboard. This represents the state-of-the-art alignment with human preferences without any manual efforts, validating DR-Arena as a reliable alternative for costly human adjudication.

Figures (8)

• Figure 1: Overview of the DR-Arena Framework.
• Figure 2: Automated Task Generation via Dynamic Information Trees.
• Figure 3: Leaderboard Correlation Analysis.
• Figure 4: Evolution of Correlation across Rounds.
• Figure 5: Relationship between Skill Gap and Rounds.