Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Learning to Summarize with Large Language Models as References

Yixin Liu, Kejian Shi, Katherine S He, Longtian Ye, Alexander R. Fabbri, Pengfei Liu, Dragomir Radev, Arman Cohan

TL;DR

This work investigates learning a smaller text summarizer under an LLM-as-reference paradigm, treating LLMs as both gold summary generators and evaluators. It introduces generation-based (MLE) and contrastive (BRIO) learning with LLM-derived supervision (GPTScore, GPTRank), and validates them on CNNDM and XSum across low- and high-resource settings. The results show that better references and contrastive learning substantially improve performance, while a meta-analysis reveals that LLM-based evaluation does not consistently align with human judgments for closely matched systems. The study highlights both the potential and challenges of integrating LLMs into the full development loop for efficient, high-quality summarization at smaller scales.

Abstract

Recent studies have found that summaries generated by large language models (LLMs) are favored by human annotators over the original reference summaries in commonly used summarization datasets. Therefore, we study an LLM-as-reference learning setting for smaller text summarization models to investigate whether their performance can be substantially improved. To this end, we use LLMs as both oracle summary generators for standard supervised fine-tuning and oracle summary evaluators for efficient contrastive learning that leverages the LLMs' supervision signals. We conduct comprehensive experiments with source news articles and find that (1) summarization models trained under the LLM-as-reference setting achieve significant performance improvement in both LLM and human evaluations; (2) contrastive learning outperforms standard supervised fine-tuning under both low and high resource settings. Our experimental results also enable a meta-analysis of LLMs' summary evaluation capacities under a challenging setting, showing that LLMs are not well-aligned with human evaluators. Particularly, our expert human evaluation reveals remaining nuanced performance gaps between LLMs and our fine-tuned models, which LLMs fail to capture. Thus, we call for further studies into both the potential and challenges of using LLMs in summarization model development.

On Learning to Summarize with Large Language Models as References

TL;DR

This work investigates learning a smaller text summarizer under an LLM-as-reference paradigm, treating LLMs as both gold summary generators and evaluators. It introduces generation-based (MLE) and contrastive (BRIO) learning with LLM-derived supervision (GPTScore, GPTRank), and validates them on CNNDM and XSum across low- and high-resource settings. The results show that better references and contrastive learning substantially improve performance, while a meta-analysis reveals that LLM-based evaluation does not consistently align with human judgments for closely matched systems. The study highlights both the potential and challenges of integrating LLMs into the full development loop for efficient, high-quality summarization at smaller scales.

Abstract

Recent studies have found that summaries generated by large language models (LLMs) are favored by human annotators over the original reference summaries in commonly used summarization datasets. Therefore, we study an LLM-as-reference learning setting for smaller text summarization models to investigate whether their performance can be substantially improved. To this end, we use LLMs as both oracle summary generators for standard supervised fine-tuning and oracle summary evaluators for efficient contrastive learning that leverages the LLMs' supervision signals. We conduct comprehensive experiments with source news articles and find that (1) summarization models trained under the LLM-as-reference setting achieve significant performance improvement in both LLM and human evaluations; (2) contrastive learning outperforms standard supervised fine-tuning under both low and high resource settings. Our experimental results also enable a meta-analysis of LLMs' summary evaluation capacities under a challenging setting, showing that LLMs are not well-aligned with human evaluators. Particularly, our expert human evaluation reveals remaining nuanced performance gaps between LLMs and our fine-tuned models, which LLMs fail to capture. Thus, we call for further studies into both the potential and challenges of using LLMs in summarization model development.
Paper Structure (50 sections, 12 equations, 4 figures, 10 tables)

This paper contains 50 sections, 12 equations, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Preliminary
  4. Large Language Models as References
  5. LLMs as Gold Summary Generators
  6. MLE Fine-tuning
  7. Contrastive Learning
  8. LLMs as Gold Summary Evaluators
  9. GPTScore for Summary Quality Evaluation
  10. GPTRank for Summary Quality Evaluation
  11. Learning with LLMs as References
  12. Learning under Low Resource Settings
  13. Experimental Setting
  14. Data Source
  15. Training Details
  16. ...and 35 more sections

Figures (4)

  • Figure 1: Pairwise comparison (GPTRank) results of different models against GPT-3.5 under GPT-3.5's evaluation (left) and GPT-4's evaluation (right). BART.GPT-3.5 and BART.GPT-4 are fine-tuned with MLE training and GPT-3.5/GPT-4 as the reference, BRIO.GPT-3.5 and BRIO.GPT-4 are fine-tuned with contrastive learning.
  • Figure 2: Results of T5 and BART models compared against GPT-3.5 under GPT-4's evaluation. BART.GPT-4 and T5.GPT-4 are MLE fine-tuned, BRIO.GPT-4 and T5BRIO.GPT-4 are fine-tuned with contrastive learning.
  • Figure 3: Results on XSum dataset. Different models are compared against GPT-3.5 under GPT-4's evaluation. BART.GPT-4 is fine-tuned with MLE training while BRIO.GPT-4 is fine-tuned with contrastive learning.
  • Figure 4: Model performance under low and high resource settings. Models are compared against GPT-3.5 under GPT-4's evaluation. The models trained with Llama-2 are under high resource settings and the models trained with GPT-4 are under low resource settings.