DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Report

TL;DR

Deep Research Bench II introduces a grounded, expert-derived benchmark for evaluating Deep Research Systems (DRS) using $132$ tasks across $22$ domains and $9{,}430$ fine-grained binary rubrics across three dimensions: information recall, analysis, and presentation. A four-stage pipeline—LLM extraction, self-evaluation, manual revision, and expert refinement—produces atomic, verifiable rubrics aligned with human judgment, enabling end-to-end rubric-based judging by an LLM. Evaluation across a diverse set of frontier models reveals a substantial gap to human experts, with even the strongest systems passing fewer than $50\%$ of rubrics and notable weaknesses in recall and analysis. The paper also investigates robustness to language and topic, leakage concerns, and future directions like user-adaptive presentation, laying a framework for more reliable, human-aligned deep-research evaluation and driving progress in DRS development.

Abstract

Deep Research Systems (DRS) aim to help users search the web, synthesize information, and deliver comprehensive investigative reports. However, how to rigorously evaluate these systems remains under-explored. Existing deep-research benchmarks often fall into two failure modes. Some do not adequately test a system's ability to analyze evidence and write coherent reports. Others rely on evaluation criteria that are either overly coarse or directly defined by LLMs (or both), leading to scores that can be biased relative to human experts and are hard to verify or interpret. To address these issues, we introduce Deep Research Bench II, a new benchmark for evaluating DRS-generated reports. It contains 132 grounded research tasks across 22 domains; for each task, a system must produce a long-form research report that is evaluated by a set of 9430 fine-grained binary rubrics in total, covering three dimensions: information recall, analysis, and presentation. All rubrics are derived from carefully selected expert-written investigative articles and are constructed through a four-stage LLM+human pipeline that combines automatic extraction with over 400 human-hours of expert review, ensuring that the criteria are atomic, verifiable, and aligned with human expert judgment. We evaluate several state-of-the-art deep-research systems on Deep Research Bench II and find that even the strongest models satisfy fewer than 50% of the rubrics, revealing a substantial gap between current DRSs and human experts.

Figures (5)

• Figure 1: Comparison of evaluation schemes in prior deep-research benchmarks and DeepResearch Bench II. Top: Benchmarks that rely on LLM-defined criteria can be misaligned with human experts. Middle: Benchmarks that adopt human-written but coarse rubrics allow seemingly correct hallucinations from the DRS to pass. Bottom: DeepResearch Bench II derives fine-grained, content-bearing rubrics from human expert reports, enabling the LLM judge to reject seemingly correct hallucinations and provide unbiased, verifiable evaluations aligned with human judgment.
• Figure 2: Topic distribution of source articles/tasks used for our benchmark.
• Figure 3: Illustration of the three-layer framework for Deep Research---information recall, analysis, and presentation.
• Figure 4: This diagram illustrates the entire workflow of our work. By decomposing human-expert articles into fine-grained, verifiable rubrics, we enable a comparison between human-expert articles and model-generated articles.
• Figure 5: Frequency distribution of rubric counts per task across the three evaluation dimensions: InfoRecall, Analysis, and Presentation. The distributions show the concentration of rubric counts within specific ranges for each dimension.