DeepResearchEval: An Automated Framework for Deep Research Task Construction and Agentic Evaluation

TL;DR

DeepResearchEval presents an automated end-to-end framework for constructing deep research tasks and evaluating agentic reports. It combines a persona-driven task creation pipeline with two filters to produce challenging, multi-source tasks, and an adaptive, task-aware evaluation pipeline supplemented by active fact-checking that verifies both cited and uncited statements. Experimental results across nine systems reveal clear strengths and weaknesses by dimension, and demonstrate the value of task-specific evaluation over fixed rubrics. The framework advances reliable long-form research evaluation and highlights practical considerations for scalability and multilingual deployment. Overall, it offers a dynamic, transparent benchmark for guiding future improvements in deep research agents.

Abstract

Deep research systems are widely used for multi-step web research, analysis, and cross-source synthesis, yet their evaluation remains challenging. Existing benchmarks often require annotation-intensive task construction, rely on static evaluation dimensions, or fail to reliably verify facts when citations are missing. To bridge these gaps, we introduce DeepResearchEval, an automated framework for deep research task construction and agentic evaluation. For task construction, we propose a persona-driven pipeline generating realistic, complex research tasks anchored in diverse user profiles, applying a two-stage filter Task Qualification and Search Necessity to retain only tasks requiring multi-source evidence integration and external retrieval. For evaluation, we propose an agentic pipeline with two components: an Adaptive Point-wise Quality Evaluation that dynamically derives task-specific evaluation dimensions, criteria, and weights conditioned on each generated task, and an Active Fact-Checking that autonomously extracts and verifies report statements via web search, even when citations are missing.

Figures (5)

• Figure 1: Overview of deep research systems' performance on our benchmark. The upper section reports quality evaluation results across deep research systems, with Gemini-2.5-Pro achieving the highest score ($8.51$/$10$). The bottom section reports factual correctness, where Manus achieves the highest ratio of correct statements ($82.3\%$).
• Figure 2: The proposed three-stage pipeline for constructing persona-driven deep research tasks.
• Figure 3: Domain Distribution and Example.
• Figure 4: Overview of the proposed pipeline. (Top) Adaptive Point-wise Quality Evaluation augments $\mathcal{D}_{\text{general}}$ with task-specific $\mathcal{D}_{\text{task}}$. An LLM scores criteria $s_{d,c}$, aggregating them into $S_{\text{quality}}$ via weights $W_d$ and $w_{d,c}$. (Bottom) Active Fact-Checking extracts statements $\mathcal{S}_i$ from report segments $\{p_i\}$. An agent verifies claims using MCP-based retrieval, producing JSON labels (Right, Wrong, Unknown).
• Figure 5: Agreement between our annotations and human experts.