DeepSynth-Eval: Objectively Evaluating Information Consolidation in Deep Survey Writing

TL;DR

DeepSynth-Eval addresses the lack of objective benchmarks for long-form post-retrieval synthesis in deep research by using high-quality surveys as gold standards. It constructs Oracle Contexts from bibliographies and derives fine-grained General and Constraint Checklists to quantify factual coverage and structural adherence, enabling objective, item-level evaluation. The benchmark comprises 96 tasks with an automated construction pipeline and a rigorous judge-based scoring system, providing Saturation-based aggregation through metrics like $S_{gen}$, $S_{con}$, and $S_{all}$. Experiments show that agentic plan-and-write workflows outperform single-turn generation but that synthesizing 100+ references remains inherently challenging, highlighting substantial room for improvement and the framework’s openness for reproducible research and RL-based improvement.

Abstract

The evolution of Large Language Models (LLMs) towards autonomous agents has catalyzed progress in Deep Research. While retrieval capabilities are well-benchmarked, the post-retrieval synthesis stage--where agents must digest massive amounts of context and consolidate fragmented evidence into coherent, long-form reports--remains under-evaluated due to the subjectivity of open-ended writing. To bridge this gap, we introduce DeepSynth-Eval, a benchmark designed to objectively evaluate information consolidation capabilities. We leverage high-quality survey papers as gold standards, reverse-engineering research requests and constructing "Oracle Contexts" from their bibliographies to isolate synthesis from retrieval noise. We propose a fine-grained evaluation protocol using General Checklists (for factual coverage) and Constraint Checklists (for structural organization), transforming subjective judgment into verifiable metrics. Experiments across 96 tasks reveal that synthesizing information from hundreds of references remains a significant challenge. Our results demonstrate that agentic plan-and-write workflows significantly outperform single-turn generation, effectively reducing hallucinations and improving adherence to complex structural constraints.

Figures (9)

• Figure 1: An illustration of DeepSynth-Eval. (a) From a reference survey, we derive a general prompt and constraint questions, and construct corresponding general/constraint checklists for evaluation. (b) The full question (prompt + constraints + reference papers) is fed to a tested LLM to generate a report. (c) A judge model verifies each checklist item on the generated report, converting subjective synthesis evaluation into objective item-wise verification.
• Figure 2: Overview of the DeepSynth-Eval construction process. Candidates generated by LLMs undergo human verification and editing to ensure quality.
• Figure 3: Distribution of checklist item counts
• Figure 4: Distribution of total citation lengths
• Figure 5: Checklist requirement status breakdown across four E2E models.