Assessing LLMs for Zero-shot Abstractive Summarization Through the Lens of Relevance Paraphrasing

TL;DR

The paper introduces relevance paraphrasing to probe the robustness of zero-shot abstractive summarization by LLMs. It isolates sentence-level salience via a proxy mapping, paraphrases key input sentences to create minimally perturbed articles, and measures how summarization quality degrades using ROUGE and BertScore across four datasets and four LLMs. Results indicate substantial variability and non-robustness in LLM summaries after perturbation, with human evaluators aligning with the metric-based degradation. The work highlights the need for task-specific robustness analyses and proposes a practical framework for evaluating and improving consistency in LLM-driven summarization.

Abstract

Large Language Models (LLMs) have achieved state-of-the-art performance at zero-shot generation of abstractive summaries for given articles. However, little is known about the robustness of such a process of zero-shot summarization. To bridge this gap, we propose relevance paraphrasing, a simple strategy that can be used to measure the robustness of LLMs as summarizers. The relevance paraphrasing approach identifies the most relevant sentences that contribute to generating an ideal summary, and then paraphrases these inputs to obtain a minimally perturbed dataset. Then, by evaluating model performance for summarization on both the original and perturbed datasets, we can assess the LLM's one aspect of robustness. We conduct extensive experiments with relevance paraphrasing on 4 diverse datasets, as well as 4 LLMs of different sizes (GPT-3.5-Turbo, Llama-2-13B, Mistral-7B, and Dolly-v2-7B). Our results indicate that LLMs are not consistent summarizers for the minimally perturbed articles, necessitating further improvements.

Figures (7)

• Figure 1: An example showcasing relevance paraphrasing. When sentences relevant to generating the summary are paraphrased to create a minimally perturbed article, we find that summarizaton performance drops as the model uses other sentences instead to craft the summary, leading to a loss of salient information.
• Figure 2: Paraphrasing results in different summaries.
• Figure 3: Summarization performance evaluation using ROUGE-1 and BertScore metrics post relevance paraphrasing.
• Figure 4: Evaluating summarization performance using ROUGE-2/L on original and paraphrased articles.
• Figure 5: Paraphrasing results in different summaries for Temperature set to 0.