Reducing Hallucinations in Entity Abstract Summarization with Facts-Template Decomposition

TL;DR

SlotSum addresses hallucinations in entity abstract summarization by disentangling the summary into a Facts component and a Template with slots. It generates templates from input documents, fills slots with predicted facts, and then corrects those facts with external knowledge, enabling explainable error localization and factual correction. The authors construct WikiFactSum by augmenting WikiSum with external facts from WikiBio/WikiAsp and demonstrate that external knowledge correction improves factual metrics and human judgments, while showing limitations when templates still harbor unaddressed facts. Overall, the approach achieves more factually correct summaries with controllable, knowledge-augmented generation, highlighting the value of template-guided fact filling for reliable entity descriptions in real-world applications.

Abstract

Entity abstract summarization aims to generate a coherent description of a given entity based on a set of relevant Internet documents. Pretrained language models (PLMs) have achieved significant success in this task, but they may suffer from hallucinations, i.e. generating non-factual information about the entity. To address this issue, we decompose the summary into two components: Facts that represent the factual information about the given entity, which PLMs are prone to fabricate; and Template that comprises generic content with designated slots for facts, which PLMs can generate competently. Based on the facts-template decomposition, we propose SlotSum, an explainable framework for entity abstract summarization. SlotSum first creates the template and then predicts the fact for each template slot based on the input documents. Benefiting from our facts-template decomposition, SlotSum can easily locate errors and further rectify hallucinated predictions with external knowledge. We construct a new dataset WikiFactSum to evaluate the performance of SlotSum. Experimental results demonstrate that SlotSum could generate summaries that are significantly more factual with credible external knowledge.

Figures (5)

• Figure 1: A demonstration of the entity abstract summarization task. The summarization model aims to generate a brief summary $S$ that resembles the golden summary $S^*$ about the person Peter Wichers from source documents $D$ collected from the Internet. Blue indicates correct facts, and Red indicates noise and hallucinated predictions.
• Figure 2: The overall framework of SlotSum, with the entity George Burroughs Torrey as an example. In step 1, SlotSum generates a template from the source documents which contain several slots to be filled like name and birth_date. In step 2, SlotSum first predicts the slots according to the source documents and then corrects hallucinated predictions (like 1903 for birth_date) with credible external knowledge. Finally, The summary is constructed by filling slots into the template.
• Figure 3: A case study on the entity Marc Muniesa Martínez. Blue texts indicate golden facts, Green texts indicate correct predictions, and Red texts indicate hallucinated predictions.
• Figure 4: Data leakage in ChatGPT, with the entity Peter Wichers and Marc Muniesa Martinez as examples.
• Figure 5: The frequency of different slots in WikiFactSum.