Can Deep Research Agents Find and Organize? Evaluating the Synthesis Gap with Expert Taxonomies

TL;DR

TaxoBench introduces a diagnostic benchmark to evaluate whether Deep Research Agents can write surveys like human experts by measuring both retrieval of core papers and expert like taxonomy construction. It builds ground truth taxonomies from 72 expert surveys totaling 3815 cited papers and provides Deep Research and Bottom Up evaluation modes. Across 7 deep research agents and 12 frontier LLMs the study reveals a major retrieval bottleneck with best recall 20.9 percent and an organization bottleneck with best ARI 0.31 even with perfect input. The results show retrieval quality strongly predicts structure quality and that providing exact papers helps organization but the gap to expert cognition remains large, underscoring the need for models to incorporate implicit domain knowledge.

Abstract

Deep Research Agents are increasingly used for automated survey generation. However, whether they can write surveys like human experts remains unclear. Existing benchmarks focus on fluency or citation accuracy, but none evaluates the core capabilities: retrieving essential papers and organizing them into coherent knowledge structures. We introduce TaxoBench, a diagnostic benchmark derived from 72 highly-cited computer science surveys. We manually extract expert-authored taxonomy trees containing 3,815 precisely categorized citations as ground truth. Our benchmark supports two evaluation modes: Deep Research mode tests end-to-end retrieval and organization given only a topic, while Bottom-Up mode isolates structuring capability by providing the exact papers human experts used. We evaluate 7 leading Deep Research agents and 12 frontier LLMs. Results reveal a dual bottleneck: the best agent recalls only 20.9% of expert-selected papers, and even with perfect input, the best model achieves only 0.31 ARI in organization. Current deep research agents remain far from expert-level survey writing. Our benchmark is publicly available at https://github.com/KongLongGeFDU/TaxoBench.

Figures (19)

• Figure 1: Deep Research Agents for survey generation. Given a survey topic, the agent autonomously conducts web-based research to retrieve relevant papers and organizes them into a hierarchical taxonomy.
• Figure 2: Overview of TaxoBench, (1) Deep Research Mode targets end-to-end retrieval and taxonomy generation ($\hat{T}$) from a query. (2) Bottom-Up Mode isolates organization logic, reconstructing hierarchies ($T^*$) from fixed papers using varying input granularities and (3) Data Example illustrates the dataset structure, detailing the ground truth JSON schema and taxonomy mapping.
• Figure 3: Retrieval performance of Deep Research agents. Bars indicate percentage of expert-selected papers retrieved.
• Figure 4: Homogeneity vs Completeness trade-off at leaf level clustering. Point size indicates ARI score.
• Figure 5: Correlation between core literature recall and taxonomy structure quality (LLM-as-Judge average). Each point represents a Deep Research agent. The dashed line shows the linear trend.