Entity-level Factual Consistency of Abstractive Text Summarization

TL;DR

The paper addresses entity-level factual consistency in abstractive summarization and the entity hallucination problem where generated summaries include named-entities not present in the source. It introduces three entity-level metrics based on Named Entity Recognition, including $prec_s = N(h ∩ s)/N(h)$, $prec_t = N(h ∩ t)/N(h)$, $recall_t = N(h ∩ t)/N(t)$, and $F1_t = 2 * prec_t * recall_t / (prec_t + recall_t)$, and a data-filtering strategy to reduce hallucination. The study shows that entity-based data filtering, plus multi-task learning with a BIO classifier and the JAENS joint generation approach, substantially improve entity-level metrics with minimal ROUGE degradation. Experiments on Newsroom, CNNDM, and XSUM with a BART-large backbone demonstrate clear gains in entity fidelity across datasets. These methods offer scalable, practical avenues to enhance factual fidelity in abstractive summarization.

Abstract

A key challenge for abstractive summarization is ensuring factual consistency of the generated summary with respect to the original document. For example, state-of-the-art models trained on existing datasets exhibit entity hallucination, generating names of entities that are not present in the source document. We propose a set of new metrics to quantify the entity-level factual consistency of generated summaries and we show that the entity hallucination problem can be alleviated by simply filtering the training data. In addition, we propose a summary-worthy entity classification task to the training process as well as a joint entity and summary generation approach, which yield further improvements in entity level metrics.