Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DUAL: Diversity and Uncertainty Active Learning for Text Summarization

Petros Stylianos Giouroukis, Alexios Gidiotis, Grigorios Tsoumakas

TL;DR

This work tackles data efficiency in abstractive text summarization by introducing DUAL, a hybrid active learning method that jointly leverages diversity-based sampling (IDDS) and uncertainty-based pruning (BAS via BLEUVar). By selecting a diverse top-k subset and then choosing the most informative sample within it, with optional random exploration, DUAL addresses both representativeness and sample quality while mitigating noisy selections and limited exploration. Across multiple Transformer-based summarization models (e.g., BART, PEGASUS, FLAN-T5) and diverse datasets (AESLC, Reddit TIFU, WikiHow, BillSum), DUAL consistently matches or outperforms the best existing AL strategies and remains robust across settings. Visual and quantitative analyses reveal why prior AL methods struggle in summarization and demonstrate that DUAL achieves a favorable balance between coverage of the data distribution and informativeness, with practical computational efficiency for real-world annotation budgets.

Abstract

With the rise of large language models, neural text summarization has advanced significantly in recent years. However, even state-of-the-art models continue to rely heavily on high-quality human-annotated data for training and evaluation. Active learning is frequently used as an effective way to collect such datasets, especially when annotation resources are scarce. Active learning methods typically prioritize either uncertainty or diversity but have shown limited effectiveness in summarization, often being outperformed by random sampling. We present Diversity and Uncertainty Active Learning (DUAL), a novel algorithm that combines uncertainty and diversity to iteratively select and annotate samples that are both representative of the data distribution and challenging for the current model. DUAL addresses the selection of noisy samples in uncertainty-based methods and the limited exploration scope of diversity-based methods. Through extensive experiments with different summarization models and benchmark datasets, we demonstrate that DUAL consistently matches or outperforms the best performing strategies. Using visualizations and quantitative metrics, we provide valuable insights into the effectiveness and robustness of different active learning strategies, in an attempt to understand why these strategies haven't performed consistently in text summarization. Finally, we show that DUAL strikes a good balance between diversity and robustness.

DUAL: Diversity and Uncertainty Active Learning for Text Summarization

TL;DR

This work tackles data efficiency in abstractive text summarization by introducing DUAL, a hybrid active learning method that jointly leverages diversity-based sampling (IDDS) and uncertainty-based pruning (BAS via BLEUVar). By selecting a diverse top-k subset and then choosing the most informative sample within it, with optional random exploration, DUAL addresses both representativeness and sample quality while mitigating noisy selections and limited exploration. Across multiple Transformer-based summarization models (e.g., BART, PEGASUS, FLAN-T5) and diverse datasets (AESLC, Reddit TIFU, WikiHow, BillSum), DUAL consistently matches or outperforms the best existing AL strategies and remains robust across settings. Visual and quantitative analyses reveal why prior AL methods struggle in summarization and demonstrate that DUAL achieves a favorable balance between coverage of the data distribution and informativeness, with practical computational efficiency for real-world annotation budgets.

Abstract

With the rise of large language models, neural text summarization has advanced significantly in recent years. However, even state-of-the-art models continue to rely heavily on high-quality human-annotated data for training and evaluation. Active learning is frequently used as an effective way to collect such datasets, especially when annotation resources are scarce. Active learning methods typically prioritize either uncertainty or diversity but have shown limited effectiveness in summarization, often being outperformed by random sampling. We present Diversity and Uncertainty Active Learning (DUAL), a novel algorithm that combines uncertainty and diversity to iteratively select and annotate samples that are both representative of the data distribution and challenging for the current model. DUAL addresses the selection of noisy samples in uncertainty-based methods and the limited exploration scope of diversity-based methods. Through extensive experiments with different summarization models and benchmark datasets, we demonstrate that DUAL consistently matches or outperforms the best performing strategies. Using visualizations and quantitative metrics, we provide valuable insights into the effectiveness and robustness of different active learning strategies, in an attempt to understand why these strategies haven't performed consistently in text summarization. Finally, we show that DUAL strikes a good balance between diversity and robustness.

Paper Structure

This paper contains 19 sections, 2 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Bayesian Active Summarization
  4. In-Domain Diversity Sampling
  5. Methodology
  6. Experimental Setup
  7. Active Learning Framework
  8. Models and Training
  9. Datasets
  10. Embeddings
  11. Results and Discussion
  12. Performance
  13. Visualization Analysis of Selected Samples
  14. Diversity and Outliers
  15. Computational Efficiency
  16. ...and 4 more sections

Figures (4)

  • Figure 1: An iteration cycle of DUAL, comprising the selection of samples, their labeling and model retraining
  • Figure 2: Rouge-1 scores of AL strategies across models and datasets
  • Figure 3: Visualization of the embedding space showing selected samples by each AL strategy in each dataset
  • Figure 4: Diversity and outlier scores for different AL strategies in each dataset