Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How Much Annotation is Needed to Compare Summarization Models?

Chantal Shaib, Joe Barrow, Alexa F. Siu, Byron C. Wallace, Ani Nenkova

TL;DR

The paper investigates how much annotation is needed to reliably compare summarization models, focusing on news-domain tasks. It shows that both automatic metrics and human judgments converge on model preferences with around $n \approx 50$ test samples, enabling efficient benchmarking. Automatic evaluations (ROUGE-1, BERTScore, SummaC-ZS, and GPT-4 as annotator) can moderately predict human preferences, with ROUGE-1 and GPT-4 performing best but still imperfectly. The work highlights that human preferences vary by downstream use-case and data source, underscoring the need for task-aware validation of automatic scores and proposing practical guidelines for minimal data in model comparison.

Abstract

Modern instruction-tuned models have become highly capable in text generation tasks such as summarization, and are expected to be released at a steady pace. In practice one may now wish to choose confidently, but with minimal effort, the best performing summarization model when applied to a new domain or purpose. In this work, we empirically investigate the test sample size necessary to select a preferred model in the context of news summarization. Empirical results reveal that comparative evaluation converges quickly for both automatic and human evaluation, with clear preferences for a system emerging from under 100 examples. The human preference data allows us to quantify how well automatic scores can reproduce preference rankings across a variety of downstream summarization tasks. We find that, while automatic metrics are stable at smaller sample sizes, only some automatic metrics are able to moderately predict model win rates according to human preference.

How Much Annotation is Needed to Compare Summarization Models?

TL;DR

The paper investigates how much annotation is needed to reliably compare summarization models, focusing on news-domain tasks. It shows that both automatic metrics and human judgments converge on model preferences with around test samples, enabling efficient benchmarking. Automatic evaluations (ROUGE-1, BERTScore, SummaC-ZS, and GPT-4 as annotator) can moderately predict human preferences, with ROUGE-1 and GPT-4 performing best but still imperfectly. The work highlights that human preferences vary by downstream use-case and data source, underscoring the need for task-aware validation of automatic scores and proposing practical guidelines for minimal data in model comparison.

Abstract

Modern instruction-tuned models have become highly capable in text generation tasks such as summarization, and are expected to be released at a steady pace. In practice one may now wish to choose confidently, but with minimal effort, the best performing summarization model when applied to a new domain or purpose. In this work, we empirically investigate the test sample size necessary to select a preferred model in the context of news summarization. Empirical results reveal that comparative evaluation converges quickly for both automatic and human evaluation, with clear preferences for a system emerging from under 100 examples. The human preference data allows us to quantify how well automatic scores can reproduce preference rankings across a variety of downstream summarization tasks. We find that, while automatic metrics are stable at smaller sample sizes, only some automatic metrics are able to moderately predict model win rates according to human preference.
Paper Structure (16 sections, 8 figures, 3 tables)

This paper contains 16 sections, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Unnecessarily Large Benchmarks
  4. Human Preferences
  5. Stability of Preference
  6. Human Preference Varies by Task and Input Source
  7. Validating Automatic Evaluation
  8. Conclusions
  9. Summarization Prompt Details
  10. BLEU and SummaC-ZS
  11. Human Evaluation Win Rates and Sample Sizes: XSUM and Newsroom
  12. Human Evaluation Win Rates and Tasks: XSUM and Newsroom
  13. Annotator Agreement on Text Quality and Factuality
  14. Annotation Details
  15. Costs
  16. ...and 1 more sections

Figures (8)

  • Figure 1: Distributions of average ROUGE-1 and BERTScores across 1000 re-samples. Differences between systems emerge clearly and quickly for XSUM and Newsroom.
  • Figure 2: Aggregated annotator win rates for the CNN/DM dataset for the overall metric. Model preferences remain fairly stable across all sample sizes except in one case for sample size of 25.
  • Figure 3: Aggregated annotator win rates across all metrics. Model preferences can change depending on the task setting.
  • Figure 4: Distributions of averaged BLEU and SummaC-ZS scores across 1000 re-samples for CNN/DM, XSUM, and Newsroom.
  • Figure 5: Win rates aggregated by annotators (XSUM).
  • ...and 3 more figures