Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Utilizing GPT to Enhance Text Summarization: A Strategy to Minimize Hallucinations

Hassan Shakil, Zeydy Ortiz, Grant C. Forbes

TL;DR

This work tackles hallucinations in AI-generated text summaries by integrating extractive, abstractive, and hybrid approaches (DistilBERT and T5) with a GPT-based refinement stage. The pipeline generates unrefined summaries, refines them via prompted GPT evaluations, and assesses improvements using a diverse metric set including FactSumm, QAGS, SummaC, ROUGE, and GPT-3.5 Turbo derived analyses. Results show significant gains in factual consistency and hallucination reduction, particularly for abstractive and hybrid summaries, though some metrics exhibit mixed responses. The study highlights the need for evaluation frameworks that better capture semantic and factual fidelity in large language model assisted summarization and proposes a practical path toward more reliable automatic summarization systems.

Abstract

In this research, we uses the DistilBERT model to generate extractive summary and the T5 model to generate abstractive summaries. Also, we generate hybrid summaries by combining both DistilBERT and T5 models. Central to our research is the implementation of GPT-based refining process to minimize the common problem of hallucinations that happens in AI-generated summaries. We evaluate unrefined summaries and, after refining, we also assess refined summaries using a range of traditional and novel metrics, demonstrating marked improvements in the accuracy and reliability of the summaries. Results highlight significant improvements in reducing hallucinatory content, thereby increasing the factual integrity of the summaries.

Utilizing GPT to Enhance Text Summarization: A Strategy to Minimize Hallucinations

TL;DR

This work tackles hallucinations in AI-generated text summaries by integrating extractive, abstractive, and hybrid approaches (DistilBERT and T5) with a GPT-based refinement stage. The pipeline generates unrefined summaries, refines them via prompted GPT evaluations, and assesses improvements using a diverse metric set including FactSumm, QAGS, SummaC, ROUGE, and GPT-3.5 Turbo derived analyses. Results show significant gains in factual consistency and hallucination reduction, particularly for abstractive and hybrid summaries, though some metrics exhibit mixed responses. The study highlights the need for evaluation frameworks that better capture semantic and factual fidelity in large language model assisted summarization and proposes a practical path toward more reliable automatic summarization systems.

Abstract

In this research, we uses the DistilBERT model to generate extractive summary and the T5 model to generate abstractive summaries. Also, we generate hybrid summaries by combining both DistilBERT and T5 models. Central to our research is the implementation of GPT-based refining process to minimize the common problem of hallucinations that happens in AI-generated summaries. We evaluate unrefined summaries and, after refining, we also assess refined summaries using a range of traditional and novel metrics, demonstrating marked improvements in the accuracy and reliability of the summaries. Results highlight significant improvements in reducing hallucinatory content, thereby increasing the factual integrity of the summaries.
Paper Structure (29 sections, 3 figures, 1 table, 2 algorithms)

This paper contains 29 sections, 3 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Methodology
  4. Generation of Unrefined Summaries
  5. Extractive Summarization Using DistilBERT
  6. Abstractive Summarization Using T5
  7. Hybrid Summarization Approach
  8. Algorithm: Hallucination Reduction in Text Summaries
  9. GPT-Based Evaluation and Refinement Process
  10. Implementation Details
  11. Evaluation Metrics
  12. FactSumm
  13. QAGS (Asking and Answering Questions)
  14. SummaC
  15. ROUGE (Recall-Oriented Understudy for Gisting Evaluation)
  16. ...and 14 more sections

Figures (3)

  • Figure 1: Schematic Representation of the Research Methodology
  • Figure 2: Comparison of evaluation metrics for unrefined and refined summaries.
  • Figure 3: Scatter plot illustrating the correlation between pre- and post-refinement scores, with the line of best fit highlighting the general trend of improvement.