Information-Theoretic Distillation for Reference-less Summarization

TL;DR

This paper tackles the problem of dependence on giant LLMs for effective summarization by introducing InfoSumm, an information-theoretic framework that learns a high-quality, reference-free summarizer from a small off-the-shelf LM. By formulating summarization as maximizing the mutual information $PMI(x;y)$ between document $x$ and its summary $y$ under a length constraint, and by employing self-supervised saliency, faithfulness, and brevity critics, the method generates a large synthetic dataset without human references. Through expert iteration, a compact 568M-parameter summarizer is distilled that competitive with, and in some cases superior to, ChatGPT and in-domain supervised models, across zero-shot, unseen-domain, and controllable summarization tasks. The approach also yields a diverse, reusable synthetic dataset and demonstrates strong controllability, suggesting a practical, scalable alternative to relying on LLMs for reference-based summarization in real-world applications.

Abstract

The current winning recipe for automatic summarization is using proprietary large-scale language models (LLMs) such as ChatGPT as is, or imitation learning from them as teacher models. While increasingly ubiquitous dependence on such large-scale language models is convenient, there remains an important question of whether small-scale models could have achieved competitive results, if we were to seek an alternative learning method -- that allows for a more cost-efficient, controllable, yet powerful summarizer. We present InfoSumm, a novel framework to distill a powerful summarizer based on the information-theoretic objective for summarization, without relying on either the LLM's capability or human-written references. To achieve this, we first propose a novel formulation of the desiderata of summarization (saliency, faithfulness and brevity) through the lens of mutual information between the original document and the summary. Based on this formulation, we start off from Pythia-2.8B as the teacher model, which is not yet capable of summarization, then self-train the model to optimize for the information-centric measures of ideal summaries. Distilling from the improved teacher, we arrive at a compact but powerful summarizer with only 568M parameters that performs competitively against ChatGPT, without ever relying on ChatGPT's capabilities. Extensive analysis demonstrates that our approach outperforms in-domain supervised models in human evaluation, let alone state-of-the-art unsupervised methods, and wins over ChatGPT in controllable summarization.

Figures (14)

• Figure 1: Overview of InfoSumm. We formulate summarization as (1) information maximization objective under a length constraint, which allows us to (2) self-train an expert teacher from only a small, off-the-shelf LM and self-supervised critics. Finally, (3) distilling from the improved teacher, we obtain a compact yet powerful summarizer without relying on an LLM already competent at summarization or human-annotated references.
• Figure 2: Human evaluation results. InfoSumm-0.5B is consistently scored higher than in-domain supervised PEGASUS, and outperforms ChatGPT with a simple re-ranking approach (best-of-10). Left: We compare InfoSumm-0.5B against baselines across 4 dimensions of summary quality. Right: We test best-of-10 approach on top of ChatGPT and InfoSumm-0.5B, by sampling 10 summaries per document and ranking them using the critic models of InfoSumm.
• Figure 2: InfoSumm better generalizes to unseen domains than human-supervised PEGASUS. We report ROUGE-L (R-L) and G-Eval (G-E) on WikiHow and Reddit domains.
• Figure 3: Summary style distribution of the distilled dataset from InfoSumm. Compared to human-authored datasets (Appendix §\ref{['app:additional_data_diversity_analysis']}), our dataset entails substantially diverse coverage of summary styles, leading to a more robust and generalizable student.
• Figure 4: Results on controllable summarization. InfoSumm-0.5B achieves better controllability across summary length, extractiveness, specificity than 5-shot prompted ChatGPT or human-supervised model.