DeepSurvey-Bench: Evaluating Academic Value of Automatically Generated Scientific Survey

TL;DR

The paper tackles the challenge of evaluating automatically generated scientific surveys by identifying the shortcomings of ground-truth reliance on surface metrics and proposing DeepSurvey-Bench, a benchmark with a three-dimension academic-value framework (information value, scholarly communication value, and research guidance value). It constructs a high-quality, academically annotated dataset of 163 surveys through a three-stage pipeline (initial collection, parsing/filtering, and human annotation) and defines seven quantifiable metrics to evaluate academic value, in addition to traditional surface-quality measures. The task is formalized as generating a survey $\mathcal{S}$ from a topic $t$ and corpus $R$, via a retrieved subset $R_s$ and outline $O$, with $\mathcal{S}=Generate(t,R_s,O)$, and the benchmark is validated by extensive experiments showing strong alignment with human judgments and revealing limitations in surface-only evaluations. The work analyzes baselines across backbones, demonstrates that high surface quality does not imply high academic value, and discusses limitations including the cost of LLM-based evaluation and the need for closed-loop verification to guide future model improvements.

Abstract

The rapid development of automated scientific survey generation technology has made it increasingly important to establish a comprehensive benchmark to evaluate the quality of generated surveys.Nearly all existing evaluation benchmarks rely on flawed selection criteria such as citation counts and structural coherence to select human-written surveys as the ground truth survey datasets, and then use surface-level metrics such as structural quality and reference relevance to evaluate generated surveys.However, these benchmarks have two key issues: (1) the ground truth survey datasets are unreliable because of a lack academic dimension annotations; (2) the evaluation metrics only focus on the surface quality of the survey such as logical coherence. Both issues lead to existing benchmarks cannot assess to evaluate their deep "academic value", such as the core research objectives and the critical analysis of different studies. To address the above problems, we propose DeepSurvey-Bench, a novel benchmark designed to comprehensively evaluate the academic value of generated surveys. Specifically, our benchmark propose a comprehensive academic value evaluation criteria covering three dimensions: informational value, scholarly communication value, and research guidance value. Based on this criteria, we construct a reliable dataset with academic value annotations, and evaluate the deep academic value of the generated surveys. Extensive experimental results demonstrate that our benchmark is highly consistent with human performance in assessing the academic value of generated surveys.

Figures (4)

• Figure 1: Comparison between existing evaluation frameworks and our proposed DeepSurvey-Bench.
• Figure 2: The dataset construction pipeline for our proposed DeepSurvey-Bench.
• Figure 3: Distribution our benchmarks across different topics. Other disciplines have fewer surveys, so we categorized them as "Other".
• Figure 4: Distribution of deficiencies in academic value among 20 low-quality surveys.