Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SurveyEval: Towards Comprehensive Evaluation of LLM-Generated Academic Surveys

Jiahao Zhao, Shuaixing Zhang, Nan Xu, Lei Wang

TL;DR

SurveyEval addresses the lack of standardized evaluation for LLM-generated academic surveys by introducing a cross-disciplinary benchmark that measures content quality, outline coherence, and reference accuracy. It combines an enhanced LLM-as-a-Judge framework with human-written references and a principle-based outline assessment, evaluated on a dataset spanning CS and STEM topics. The experiments show dedicated survey-generation systems outperform general and paper-writing systems, especially in STEM domains, with citation grounding and outline quality serving as key differentiators. This benchmark offers a scalable tool for diagnosing weaknesses and driving improvements in automated survey generation across diverse disciplines.

Abstract

LLM-based automatic survey systems are transforming how users acquire information from the web by integrating retrieval, organization, and content synthesis into end-to-end generation pipelines. While recent works focus on developing new generation pipelines, how to evaluate such complex systems remains a significant challenge. To this end, we introduce SurveyEval, a comprehensive benchmark that evaluates automatically generated surveys across three dimensions: overall quality, outline coherence, and reference accuracy. We extend the evaluation across 7 subjects and augment the LLM-as-a-Judge framework with human references to strengthen evaluation-human alignment. Evaluation results show that while general long-text or paper-writing systems tend to produce lower-quality surveys, specialized survey-generation systems are able to deliver substantially higher-quality results. We envision SurveyEval as a scalable testbed to understand and improve automatic survey systems across diverse subjects and evaluation criteria.

SurveyEval: Towards Comprehensive Evaluation of LLM-Generated Academic Surveys

TL;DR

SurveyEval addresses the lack of standardized evaluation for LLM-generated academic surveys by introducing a cross-disciplinary benchmark that measures content quality, outline coherence, and reference accuracy. It combines an enhanced LLM-as-a-Judge framework with human-written references and a principle-based outline assessment, evaluated on a dataset spanning CS and STEM topics. The experiments show dedicated survey-generation systems outperform general and paper-writing systems, especially in STEM domains, with citation grounding and outline quality serving as key differentiators. This benchmark offers a scalable tool for diagnosing weaknesses and driving improvements in automated survey generation across diverse disciplines.

Abstract

LLM-based automatic survey systems are transforming how users acquire information from the web by integrating retrieval, organization, and content synthesis into end-to-end generation pipelines. While recent works focus on developing new generation pipelines, how to evaluate such complex systems remains a significant challenge. To this end, we introduce SurveyEval, a comprehensive benchmark that evaluates automatically generated surveys across three dimensions: overall quality, outline coherence, and reference accuracy. We extend the evaluation across 7 subjects and augment the LLM-as-a-Judge framework with human references to strengthen evaluation-human alignment. Evaluation results show that while general long-text or paper-writing systems tend to produce lower-quality surveys, specialized survey-generation systems are able to deliver substantially higher-quality results. We envision SurveyEval as a scalable testbed to understand and improve automatic survey systems across diverse subjects and evaluation criteria.

Paper Structure

This paper contains 17 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. SurveyEval Benchmark
  3. Dataset Construction
  4. Evaluation Metrics
  5. Overall
  6. Outline
  7. Reference
  8. LLM-as-a-Judge
  9. LLM-as-a-Judge with Human-Written Reference
  10. LLM-as-a-Judge with Principle
  11. Experiments
  12. Experimental Setup
  13. Main Results
  14. Overall Evaluation.
  15. Outline Evaluation.
  16. ...and 2 more sections

Figures (1)

  • Figure 1: The overall framework of SurveyEval, which evaluates LLM-generated surveys across seven domains through content quality, outline coherence, and reference accuracy.