Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Role of Summary Content Units in Text Summarization Evaluation

Marcel Nawrath, Agnieszka Nowak, Tristan Ratz, Danilo C. Walenta, Juri Opitz, Leonardo F. R. Ribeiro, João Sedoc, Daniel Deutsch, Simon Mille, Yixin Liu, Lining Zhang, Sebastian Gehrmann, Saad Mahamood, Miruna Clinciu, Khyathi Chandu, Yufang Hou

TL;DR

The paper investigates automating Pyramid evaluation by proposing two SCU approximations: SMUs derived from AMR graphs and SGUs generated by large language models. Experiments show SGUs generally provide the strongest approximation to human SCUs, though simple baselines like sentence splitting can competitively rank systems for longer summaries. The findings suggest that full SCU generation may be unnecessary for some downstream tasks, while SGUs offer robust performance across system- and summary-level evaluation. The work highlights limitations in AMR-based SMUs and emphasizes the importance of NLI tuning for reliable automated Pyramid scoring, with implications for scalable evaluation of generated summaries.

Abstract

At the heart of the Pyramid evaluation method for text summarization lie human written summary content units (SCUs). These SCUs are concise sentences that decompose a summary into small facts. Such SCUs can be used to judge the quality of a candidate summary, possibly partially automated via natural language inference (NLI) systems. Interestingly, with the aim to fully automate the Pyramid evaluation, Zhang and Bansal (2021) show that SCUs can be approximated by automatically generated semantic role triplets (STUs). However, several questions currently lack answers, in particular: i) Are there other ways of approximating SCUs that can offer advantages? ii) Under which conditions are SCUs (or their approximations) offering the most value? In this work, we examine two novel strategies to approximate SCUs: generating SCU approximations from AMR meaning representations (SMUs) and from large language models (SGUs), respectively. We find that while STUs and SMUs are competitive, the best approximation quality is achieved by SGUs. We also show through a simple sentence-decomposition baseline (SSUs) that SCUs (and their approximations) offer the most value when ranking short summaries, but may not help as much when ranking systems or longer summaries.

On the Role of Summary Content Units in Text Summarization Evaluation

TL;DR

The paper investigates automating Pyramid evaluation by proposing two SCU approximations: SMUs derived from AMR graphs and SGUs generated by large language models. Experiments show SGUs generally provide the strongest approximation to human SCUs, though simple baselines like sentence splitting can competitively rank systems for longer summaries. The findings suggest that full SCU generation may be unnecessary for some downstream tasks, while SGUs offer robust performance across system- and summary-level evaluation. The work highlights limitations in AMR-based SMUs and emphasizes the importance of NLI tuning for reliable automated Pyramid scoring, with implications for scalable evaluation of generated summaries.

Abstract

At the heart of the Pyramid evaluation method for text summarization lie human written summary content units (SCUs). These SCUs are concise sentences that decompose a summary into small facts. Such SCUs can be used to judge the quality of a candidate summary, possibly partially automated via natural language inference (NLI) systems. Interestingly, with the aim to fully automate the Pyramid evaluation, Zhang and Bansal (2021) show that SCUs can be approximated by automatically generated semantic role triplets (STUs). However, several questions currently lack answers, in particular: i) Are there other ways of approximating SCUs that can offer advantages? ii) Under which conditions are SCUs (or their approximations) offering the most value? In this work, we examine two novel strategies to approximate SCUs: generating SCU approximations from AMR meaning representations (SMUs) and from large language models (SGUs), respectively. We find that while STUs and SMUs are competitive, the best approximation quality is achieved by SGUs. We also show through a simple sentence-decomposition baseline (SSUs) that SCUs (and their approximations) offer the most value when ranking short summaries, but may not help as much when ranking systems or longer summaries.
Paper Structure (32 sections, 7 equations, 3 figures, 3 tables)

This paper contains 32 sections, 7 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Pyramid Method.
  4. Automation of the Pyramid Method.
  5. SCU approximation I: SMU from AMR
  6. SCU approximation II: SGU from LLM
  7. Experiments
  8. Dataset and NLI models
  9. Datasets.
  10. NLI Models.
  11. Baselines
  12. STUs
  13. Sentence splitting
  14. N-grams
  15. Intrinsic Evaluation
  16. ...and 17 more sections

Figures (3)

  • Figure 1: The AMR graph for the sentence "Godfrey Elfwick recruited via Twitter to appear on World Have Your Say"
  • Figure 2: Two AMR sub-graphs for the sentence "Godfrey Elfwick recruited via Twitter to appear on World Have Your Say"
  • Figure 3: Human evaluation results. Each bar represents the sum of scores aggregated over all annotators. Upper-bound indicates the best possible result (each annotator always assigns the maximum quality score).