DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agents

TL;DR

DeepSearchQA tackles the Comprehensiveness Gap by introducing an exhaustive, set-based benchmark for deep research agents. It pairs 900 time-anchored prompts across 17 domains with ground-truth answer sets and a causal-chain task design that requires systematic search, entity de-duplication, and dynamic stopping criteria. Evaluation combines continuous F1-based metrics with strict set-based classifications, enabled by an automated LLM judge and a public Kaggle leaderboard. Results show current state-of-the-art models struggle to balance recall and precision, revealing a Last Mile gap between F1 and fully correct, and demonstrating that more compute yields non-linear gains, thus motivating architecture-level innovations for robust, autonomous, deep-research agents.

Abstract

We introduce DeepSearchQA, a 900-prompt benchmark for evaluating agents on difficult multi-step information-seeking tasks across 17 different fields. Unlike traditional benchmarks that target single answer retrieval or broad-spectrum factuality, DeepSearchQA features a dataset of challenging, handcrafted tasks designed to evaluate an agent's ability to execute complex search plans to generate exhaustive answer lists. This shift in design explicitly tests three critical, yet under-evaluated capabilities: 1) systematic collation of fragmented information from disparate sources, 2) de-duplication and entity resolution to ensure precision, and 3) the ability to reason about stopping criteria within an open-ended search space. Each task is structured as a causal chain, where discovering information for one step is dependent on the successful completion of the previous one, stressing long-horizon planning and context retention. All tasks are grounded in the open web with objectively verifiable answer sets. Our comprehensive evaluation of state-of-the-art agent architectures reveals significant performance limitations: even the most advanced models struggle to balance high recall with precision. We observe distinct failure modes ranging from premature stopping (under-retrieval) to hedging behaviors, where agents cast an overly wide net of low-confidence answers to artificially boost recall. These findings highlight critical headroom in current agent designs and position DeepSearchQA as an essential diagnostic tool for driving future research toward more robust, deep-research capabilities.

Figures (1)

• Figure 1: DeepSearchQA benchmark overview. The benchmark features a balanced distribution of prompts across diverse topics (Left), preventing domain overfitting. When evaluated on this diverse set, the Gemini Deep Research agent demonstrates strong performance scaling (Right), with accuracy increasing monotonically as more test-time compute (samples) is applied.