Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Not too long do read: Evaluating LLM-generated extreme scientific summaries

Zhuoqi Lyu, Qing Ke

TL;DR

BiomedTLDR introduces a large-scale, author-authored TLDR dataset derived from annotated bibliographies to enable scalable evaluation of LLMs on extreme scientific summarization.Using five open-weight LLMs and a few-shot prompting setup, the study compares AI-generated TLDRs with human-authored ones across descriptive statistics, readability, abstractiveness, rhetorical structure, and reference-based metrics.Findings show LLMs are generally more extractive, with higher fidelity to source text and lower abstractive novelty than humans; some models approach human-level salience identification but fail to match readability and paraphrasing quality.Distillation and model selection influence performance; the results underscore the need for human-in-the-loop workflows and a generalizable, bibliography-based data collection paradigm for cross-domain AI-assisted science communication.Overall, BiomedTLDR establishes a new benchmark for extreme scientific summarization and provides insights into how open-weight LLMs perform on domain-specific summarization tasks.

Abstract

High-quality scientific extreme summary (TLDR) facilitates effective science communication. How do large language models (LLMs) perform in generating them? How are LLM-generated summaries different from those written by human experts? However, the lack of a comprehensive, high-quality scientific TLDR dataset hinders both the development and evaluation of LLMs' summarization ability. To address these, we propose a novel dataset, BiomedTLDR, containing a large sample of researcher-authored summaries from scientific papers, which leverages the common practice of including authors' comments alongside bibliography items. We then test popular open-weight LLMs for generating TLDRs based on abstracts. Our analysis reveals that, although some of them successfully produce humanoid summaries, LLMs generally exhibit a greater affinity for the original text's lexical choices and rhetorical structures, hence tend to be more extractive rather than abstractive in general, compared to humans. Our code and datasets are available at https://github.com/netknowledge/LLM_summarization (Lyu and Ke, 2025).

Not too long do read: Evaluating LLM-generated extreme scientific summaries

TL;DR

BiomedTLDR introduces a large-scale, author-authored TLDR dataset derived from annotated bibliographies to enable scalable evaluation of LLMs on extreme scientific summarization.Using five open-weight LLMs and a few-shot prompting setup, the study compares AI-generated TLDRs with human-authored ones across descriptive statistics, readability, abstractiveness, rhetorical structure, and reference-based metrics.Findings show LLMs are generally more extractive, with higher fidelity to source text and lower abstractive novelty than humans; some models approach human-level salience identification but fail to match readability and paraphrasing quality.Distillation and model selection influence performance; the results underscore the need for human-in-the-loop workflows and a generalizable, bibliography-based data collection paradigm for cross-domain AI-assisted science communication.Overall, BiomedTLDR establishes a new benchmark for extreme scientific summarization and provides insights into how open-weight LLMs perform on domain-specific summarization tasks.

Abstract

High-quality scientific extreme summary (TLDR) facilitates effective science communication. How do large language models (LLMs) perform in generating them? How are LLM-generated summaries different from those written by human experts? However, the lack of a comprehensive, high-quality scientific TLDR dataset hinders both the development and evaluation of LLMs' summarization ability. To address these, we propose a novel dataset, BiomedTLDR, containing a large sample of researcher-authored summaries from scientific papers, which leverages the common practice of including authors' comments alongside bibliography items. We then test popular open-weight LLMs for generating TLDRs based on abstracts. Our analysis reveals that, although some of them successfully produce humanoid summaries, LLMs generally exhibit a greater affinity for the original text's lexical choices and rhetorical structures, hence tend to be more extractive rather than abstractive in general, compared to humans. Our code and datasets are available at https://github.com/netknowledge/LLM_summarization (Lyu and Ke, 2025).
Paper Structure (24 sections, 1 equation, 8 figures, 5 tables)

This paper contains 24 sections, 1 equation, 8 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Our Dataset
  4. Generating Summary using LLMs
  5. Evaluation
  6. Descriptive Statistics
  7. Readability
  8. Abstractiveness
  9. Rhetorical Structure
  10. Reference-based Evaluation
  11. Discussion
  12. Conclusion
  13. Dataset Statistics
  14. Details about Table \ref{['tab:tldr-data']}
  15. Discipline distribution
  16. ...and 9 more sections

Figures (8)

  • Figure 1: (A) An example annotated bibliography entry cited by andersson2001genetic. Bold text at the end is the annotation from the author, which is used as the summary in this research. (B) The corresponding abstract-summary pair to (A). The abstract is from riquet1999finemapping.
  • Figure 2: Word rhetorical role distribution across different types of summaries, and original abstracts as the reference. The $x$-axis represents normalized word position in texts, and color intensity indicates the average probability of belonging to each rhetorical role, based on word co-occurrence statistics against the original abstract.
  • Figure 3: The distribution of source abstracts' discipline in BiomedTLDR.
  • Figure 4: The distribution of source abstracts' discipline in CiteSum
  • Figure 5: The distribution of source abstracts' discipline in Multi-XScience
  • ...and 3 more figures