Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Total Recall QA: A Verifiable Evaluation Suite for Deep Research Agents

Mahta Rafiee, Heydar Soudani, Zahra Abbasiantaeb, Mohammad Aliannejadi, Faegheh Hasibi, Hamed Zamani

Abstract

Deep research agents have emerged as LLM-based systems designed to perform multi-step information seeking and reasoning over large, open-domain sources to answer complex questions by synthesizing information from multiple information sources. Given the complexity of the task and despite various recent efforts, evaluation of deep research agents remains fundamentally challenging. This paper identifies a list of requirements and optional properties for evaluating deep research agents. We observe that existing benchmarks do not satisfy all identified requirements. Inspired by prior research on TREC Total Recall Tracks, we introduce the task of Total Recall Question Answering and develop a framework for deep research agents evaluation that satisfies the identified criteria. Our framework constructs single-answer, total recall queries with precise evaluation and relevance judgments derived from a structured knowledge base paired with a text corpus, enabling large-scale data construction. Using this framework, we build TRQA, a deep research benchmark constructed from Wikidata-Wikipedia as a real-world source and a synthetically generated e-commerce knowledge base and corpus to mitigate the effects of data contamination. We benchmark the collection with representative retriever and deep research models and establish baseline retrieval and end-to-end results for future comparative evaluation.

Total Recall QA: A Verifiable Evaluation Suite for Deep Research Agents

Abstract

Deep research agents have emerged as LLM-based systems designed to perform multi-step information seeking and reasoning over large, open-domain sources to answer complex questions by synthesizing information from multiple information sources. Given the complexity of the task and despite various recent efforts, evaluation of deep research agents remains fundamentally challenging. This paper identifies a list of requirements and optional properties for evaluating deep research agents. We observe that existing benchmarks do not satisfy all identified requirements. Inspired by prior research on TREC Total Recall Tracks, we introduce the task of Total Recall Question Answering and develop a framework for deep research agents evaluation that satisfies the identified criteria. Our framework constructs single-answer, total recall queries with precise evaluation and relevance judgments derived from a structured knowledge base paired with a text corpus, enabling large-scale data construction. Using this framework, we build TRQA, a deep research benchmark constructed from Wikidata-Wikipedia as a real-world source and a synthetically generated e-commerce knowledge base and corpus to mitigate the effects of data contamination. We benchmark the collection with representative retriever and deep research models and establish baseline retrieval and end-to-end results for future comparative evaluation.
Paper Structure (9 sections, 4 figures, 7 tables)

This paper contains 9 sections, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. TRQA Framework and Data Construction
  3. The Data Generation Framework
  4. Real Data: Wikidata-Wikipedia
  5. Synthetic Data: E-Commerce
  6. TRQA Statistics
  7. Experimental Setup
  8. Results
  9. Conclusions

Figures (4)

  • Figure 1: An example from the TRQA benchmark.
  • Figure 2: An overview of the TRQA data generation framework for Total Recall QA tasks.
  • Figure 3: Query topic popularity across logarithmic popularity bins in TRQA-Wiki1 (right) and TRQA-Wiki2 (left).
  • Figure 4: Analysis of sub-queries issued by DRAs: (a) average number of sub-queries issued plotted against number of query gold entities, (b) average number of newly retrieved gold and distracting entities as more sub-queries are issued.