Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Benchmark for Deep Information Synthesis

Debjit Paul, Daniel Murphy, Milan Gritta, Ronald Cardenas, Victor Prokhorov, Lena Sophia Bolliger, Aysim Toker, Roy Miles, Andreea-Maria Oncescu, Jasivan Alex Sivakumar, Philipp Borchert, Ismail Elezi, Meiru Zhang, Ka Yiu Lee, Guchun Zhang, Jun Wang, Gerasimos Lampouras

TL;DR

DEEPSYNTH is a novel benchmark designed to evaluate agents on realistic, time-consuming problems that combine information gathering, synthesis, and structured reasoning to produce insights, and reveals that current agents struggle with hallucinations and reasoning over large information spaces.

Abstract

Large language model (LLM)-based agents are increasingly used to solve complex tasks involving tool use, such as web browsing, code execution, and data analysis. However, current evaluation benchmarks do not adequately assess their ability to solve real-world tasks that require synthesizing information from multiple sources and inferring insights beyond simple fact retrieval. To address this, we introduce DEEPSYNTH, a novel benchmark designed to evaluate agents on realistic, time-consuming problems that combine information gathering, synthesis, and structured reasoning to produce insights. DEEPSYNTH contains 120 tasks collected across 7 domains and data sources covering 67 countries. DEEPSYNTH is constructed using a multi-stage data collection pipeline that requires annotators to collect official data sources, create hypotheses, perform manual analysis, and design tasks with verifiable answers. When evaluated on DEEPSYNTH, 11 state-of-the-art LLMs and deep research agents achieve a maximum F1 score of 8.97 and 17.5 on the LLM-judge metric, underscoring the difficulty of the benchmark. Our analysis reveals that current agents struggle with hallucinations and reasoning over large information spaces, highlighting DEEPSYNTH as a crucial benchmark for guiding future research.

A Benchmark for Deep Information Synthesis

TL;DR

DEEPSYNTH is a novel benchmark designed to evaluate agents on realistic, time-consuming problems that combine information gathering, synthesis, and structured reasoning to produce insights, and reveals that current agents struggle with hallucinations and reasoning over large information spaces.

Abstract

Large language model (LLM)-based agents are increasingly used to solve complex tasks involving tool use, such as web browsing, code execution, and data analysis. However, current evaluation benchmarks do not adequately assess their ability to solve real-world tasks that require synthesizing information from multiple sources and inferring insights beyond simple fact retrieval. To address this, we introduce DEEPSYNTH, a novel benchmark designed to evaluate agents on realistic, time-consuming problems that combine information gathering, synthesis, and structured reasoning to produce insights. DEEPSYNTH contains 120 tasks collected across 7 domains and data sources covering 67 countries. DEEPSYNTH is constructed using a multi-stage data collection pipeline that requires annotators to collect official data sources, create hypotheses, perform manual analysis, and design tasks with verifiable answers. When evaluated on DEEPSYNTH, 11 state-of-the-art LLMs and deep research agents achieve a maximum F1 score of 8.97 and 17.5 on the LLM-judge metric, underscoring the difficulty of the benchmark. Our analysis reveals that current agents struggle with hallucinations and reasoning over large information spaces, highlighting DEEPSYNTH as a crucial benchmark for guiding future research.
Paper Structure (37 sections, 12 figures, 14 tables)

This paper contains 37 sections, 12 figures, 14 tables.

Table of Contents

  1. Introduction
  2. The DeepSynth Benchmark
  3. Criteria for DeepSynth Tasks
  4. Data Collection
  5. Data Source Identification.
  6. Hypothesis Generation.
  7. Hypothesis Validation.
  8. Task Formulation.
  9. Data Validation.
  10. Data Statistics
  11. Evaluation Setup
  12. Models.
  13. Metrics.
  14. Results
  15. Main Results
  16. ...and 22 more sections

Figures (12)

  • Figure 1: A sample task from DeepSynth, illustrating that synthesizing knowledge requires agents to perform multiple steps, including web browsing, gathering information from multiple sources, reasoning over them, and generating the final answer.
  • Figure 1: DeepSynth statistics across tasks.
  • Figure 2: An overview of our data collection process for building the DeepSynth benchmark
  • Figure 3: Percentage of tasks per capabilities required to solve DeepSynth.
  • Figure 4: Performance comparison on the DeepSynth-Dev benchmark. (a)F1 Score measures the quality of model predictions. LLM-Judge (GPT-4.1) reports the average precision as judged by GPT-4.1. Light bars denote LLM baselines; dark bars denote framework-based agents. (b) Best-of-N LLM Judge Scores comparing GPT-4.1 standalone vs. Smolagents (GPT-4.1) across $N \in \{1, 3, 5\}$ and and Self-Consistency@5 (majority voting).
  • ...and 7 more figures