Why Your Deep Research Agent Fails? On Hallucination Evaluation in Full Research Trajectory

TL;DR

Deep Research Agents currently fail to reach robust reliability because end-to-end benchmarks miss critical intermediate hallucinations along a full research trajectory. The authors introduce the PIES Taxonomy to categorize hallucinations by Planning vs. Summarization and Explicit vs. Implicit, and build DeepHalluBench to stress-test DRAs across 100 queries and adversarial no-answer cases. They develop a trajectory data pipeline and atomic decomposition, with a four-part evaluation framework covering claim verification, noise detection, action verification, and restriction checking, plus reliability metrics. Their large-scale evaluation across six DRAs reveals pervasive failures due to hallucination propagation and cognitive biases such as the Anchor Effect and Homogeneity Bias, underscoring the need for architectural fixes focused on early error correction and long-context debiasing.

Abstract

Diagnosing the failure mechanisms of Deep Research Agents (DRAs) remains a critical challenge. Existing benchmarks predominantly rely on end-to-end evaluation, obscuring critical intermediate hallucinations, such as flawed planning, that accumulate throughout the research trajectory. To bridge this gap, we propose a shift from outcome-based to process-aware evaluation by auditing the full research trajectory. We introduce the PIES Taxonomy to categorize hallucinations along functional components (Planning vs. Summarization) and error properties (Explicit vs. Implicit). We instantiate this taxonomy into a fine-grained evaluation framework that decomposes the trajectory to rigorously quantify these hallucinations. Leveraging this framework to isolate 100 distinctively hallucination-prone tasks including adversarial scenarios, we curate DeepHalluBench. Experiments on six state-of-theart DRAs reveal that no system achieves robust reliability. Furthermore, our diagnostic analysis traces the etiology of these failures to systemic deficits, specifically hallucination propagation and cognitive biases, providing foundational insights to guide future architectural optimization. Data and code are available at https://github.com/yuhao-zhan/DeepHalluBench.

Figures (18)

• Figure 1: Comparison between existing benchmarks for DRAs and our benchmark.
• Figure 2: The PIES Taxonomy. The framework intersects functional components (vertical axis) with error properties (horizontal axis). The four quadrants represent specific hallucination categories derived from these combinations: Explicit Summarization, Implicit Summarization, Explicit Planning, and Implicit Planning.
• Figure 3: The Data Acquisition and Decomposition Pipeline. We first employ custom parsers to structure raw Web UI traces into iterative plan-search-summarize loops. These loops are further decomposed by LLMs atomically to enable fine-grained evaluation.
• Figure 4: The Evaluation Framework for Summarization Hallucinations. The pipeline assesses Explicit errors (top) and Implicit neglect (bottom). The addition symbols ($\oplus$) define the data scope: selecting evidence scope for verification (top) or specifying document sets for global/local level (bottom). The cross symbol ($\otimes$) intersects ranked clusters with Chunk Memory to classify them as utilized (In-Memory) or ignored (Out-Memory), enabling the penalty quantification shown on the right.
• Figure 5: The Evaluation Framework for Planning Hallucinations. The pipeline assesses Explicit errors (top) and Implicit neglect (bottom). The subtraction symbol ($\ominus$) defines the neglect identification logic: removing the set of effectively executed sub-queries from the full set of sub-queries to isolate neglected restrictions.