Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Source Identification in Abstractive Summarization

Yoshi Suhara, Dimitris Alikaniotis

TL;DR

The paper tackles the opacity of abstractive summarization by defining source sentences—input sentences that carry essential information for a summary—and introduces SourceSum, a dataset with annotations for reference and PEGASUS-generated summaries on XSum and CNN/DM. It formulates automatic source sentence detection and compares multiple methods, including similarity-based, cross-attention, and a novel perplexity-gain approach. The results show the perplexity-gain method excels in highly abstractive settings (XSum), while similarity-based methods perform robustly in extractive settings (CNN/DM), highlighting model-dependence in source identification. This work advances interpretability in summarization and provides a benchmark for evaluating how models extract and assemble information from source texts, with code and data publicly available.

Abstract

Neural abstractive summarization models make summaries in an end-to-end manner, and little is known about how the source information is actually converted into summaries. In this paper, we define input sentences that contain essential information in the generated summary as $\textit{source sentences}$ and study how abstractive summaries are made by analyzing the source sentences. To this end, we annotate source sentences for reference summaries and system summaries generated by PEGASUS on document-summary pairs sampled from the CNN/DailyMail and XSum datasets. We also formulate automatic source sentence detection and compare multiple methods to establish a strong baseline for the task. Experimental results show that the perplexity-based method performs well in highly abstractive settings, while similarity-based methods perform robustly in relatively extractive settings. Our code and data are available at https://github.com/suhara/sourcesum.

Source Identification in Abstractive Summarization

TL;DR

The paper tackles the opacity of abstractive summarization by defining source sentences—input sentences that carry essential information for a summary—and introduces SourceSum, a dataset with annotations for reference and PEGASUS-generated summaries on XSum and CNN/DM. It formulates automatic source sentence detection and compares multiple methods, including similarity-based, cross-attention, and a novel perplexity-gain approach. The results show the perplexity-gain method excels in highly abstractive settings (XSum), while similarity-based methods perform robustly in extractive settings (CNN/DM), highlighting model-dependence in source identification. This work advances interpretability in summarization and provides a benchmark for evaluating how models extract and assemble information from source texts, with code and data publicly available.

Abstract

Neural abstractive summarization models make summaries in an end-to-end manner, and little is known about how the source information is actually converted into summaries. In this paper, we define input sentences that contain essential information in the generated summary as and study how abstractive summaries are made by analyzing the source sentences. To this end, we annotate source sentences for reference summaries and system summaries generated by PEGASUS on document-summary pairs sampled from the CNN/DailyMail and XSum datasets. We also formulate automatic source sentence detection and compare multiple methods to establish a strong baseline for the task. Experimental results show that the perplexity-based method performs well in highly abstractive settings, while similarity-based methods perform robustly in relatively extractive settings. Our code and data are available at https://github.com/suhara/sourcesum.
Paper Structure (28 sections, 3 equations, 5 figures, 9 tables)

This paper contains 28 sections, 3 equations, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. SourceSum
  3. Corpus creation
  4. Souce sentence annotation
  5. Reconstructability annotation
  6. Dataset Statistics
  7. Source Sentence Detection
  8. Problem Formulation
  9. Similarity-based Method
  10. Cross-attention Weights
  11. Perplexity Gain
  12. Evaluation
  13. Evaluation metrics
  14. Results
  15. Analysis
  16. ...and 13 more sections

Figures (5)

  • Figure 1: Annotation flow for SourceSum. For each document-summary pair, the human annotator is asked to annotate each sentence (Q1), followed by the reconstructability question (Q2).
  • Figure 2: Distribution of the number of (ground-truth) source sentences. Generated summaries tend to have more source sentences on XSum while having fewer source sentences on CNN/DM.
  • Figure 3: Correlation analysis of the source sentence detection methods.
  • Figure 4: Distribution of source sentence absolute positions. Both plots support that a commonly used lead-3 .
  • Figure 5: Distribution of the sentence interval between adjacent (ground-truth) source sentences. For example, if source sentence positions are 1, 3, and 7, the sentence intervals for the example are 2 and 4.