Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SummIt: Iterative Text Summarization via ChatGPT

Haopeng Zhang, Xiao Liu, Jiawei Zhang

TL;DR

<3-5 sentence high-level summary> SummIt introduces an iterative framework for text summarization that refines outputs through self-evaluation and feedback, avoiding supervised training. By separating a summarizer and an evaluator and enabling in-context learning, OpenIE-based knowledge extraction, and topic-focused controls, it achieves improvements in faithfulness and user-controllability across CNN/DM, XSum, and NEWTS, as validated by automatic metrics and human judgments. A notable finding is an over-correction tendency where refinements follow the evaluator's criteria more than human judgment, suggesting avenues for human-in-the-loop strategies. Overall, SummIt offers a practical, training-free approach to producing more faithful, controllable summaries through iterative refinement with LLMs.

Abstract

Text summarization systems have made significant progress in recent years, but typically generate summaries in one single step. However, the one-shot summarization setting is sometimes inadequate, as the generated summary may contain hallucinations or overlook essential details related to the reader's interests. This paper addresses this limitation by proposing SummIt, an iterative text summarization framework based on large language models like ChatGPT. Our framework enables the model to refine the generated summary iteratively through self-evaluation and feedback, resembling humans' iterative process when drafting and revising summaries. Furthermore, we explore the potential benefits of integrating knowledge and topic extractors into the framework to enhance summary faithfulness and controllability. We automatically evaluate the performance of our framework on three benchmark summarization datasets. We also conduct a human evaluation to validate the effectiveness of the iterative refinements and identify a potential issue of over-correction.

SummIt: Iterative Text Summarization via ChatGPT

TL;DR

<3-5 sentence high-level summary> SummIt introduces an iterative framework for text summarization that refines outputs through self-evaluation and feedback, avoiding supervised training. By separating a summarizer and an evaluator and enabling in-context learning, OpenIE-based knowledge extraction, and topic-focused controls, it achieves improvements in faithfulness and user-controllability across CNN/DM, XSum, and NEWTS, as validated by automatic metrics and human judgments. A notable finding is an over-correction tendency where refinements follow the evaluator's criteria more than human judgment, suggesting avenues for human-in-the-loop strategies. Overall, SummIt offers a practical, training-free approach to producing more faithful, controllable summaries through iterative refinement with LLMs.

Abstract

Text summarization systems have made significant progress in recent years, but typically generate summaries in one single step. However, the one-shot summarization setting is sometimes inadequate, as the generated summary may contain hallucinations or overlook essential details related to the reader's interests. This paper addresses this limitation by proposing SummIt, an iterative text summarization framework based on large language models like ChatGPT. Our framework enables the model to refine the generated summary iteratively through self-evaluation and feedback, resembling humans' iterative process when drafting and revising summaries. Furthermore, we explore the potential benefits of integrating knowledge and topic extractors into the framework to enhance summary faithfulness and controllability. We automatically evaluate the performance of our framework on three benchmark summarization datasets. We also conduct a human evaluation to validate the effectiveness of the iterative refinements and identify a potential issue of over-correction.
Paper Structure (31 sections, 1 equation, 3 figures, 10 tables)

This paper contains 31 sections, 1 equation, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text Summarization
  4. Summarization with Large Language Models
  5. Text Editing
  6. Methods
  7. Iterative Summarization
  8. Summarizer:
  9. Evaluator:
  10. Stopping Criteria:
  11. In-context Learning
  12. Summary Faithfulness and Controllability
  13. Knowledge Extractor:
  14. Topic Extractor:
  15. Prompt Format
  16. ...and 16 more sections

Figures (3)

  • Figure 1: An illustration of the iterative summarization process. The summarizer continuously refines the summary according to self-feedback from the evaluator at each iteration.
  • Figure 2: The overall framework of our proposed iterative text summarization system. The evaluator generates an evaluation rationale based on the current summary, and the summarizer then refines the summary accordingly. The knowledge and topic extractors retrieve information from the source document to guide the process.
  • Figure 3: Human evaluation to justify the refinement behavior of SummIt. The top plot refers to the human justification of the ratio that the summary is improved at each iteration and the bottom plot indicates the ratio that the summarizer follows the evaluator's evaluation rationale.