Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FIZZ: Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Document

Joonho Yang, Seunghyun Yoon, Byeongjeong Kim, Hwanhee Lee

TL;DR

Experimental results demonstrate that the proposed factual consistency checking system significantly outperforms existing systems and is based on fine-grained atomic facts decomposition.

Abstract

Through the advent of pre-trained language models, there have been notable advancements in abstractive summarization systems. Simultaneously, a considerable number of novel methods for evaluating factual consistency in abstractive summarization systems has been developed. But these evaluation approaches incorporate substantial limitations, especially on refinement and interpretability. In this work, we propose highly effective and interpretable factual inconsistency detection method metric Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Document for abstractive summarization systems that is based on fine-grained atomic facts decomposition. Moreover, we align atomic facts decomposed from the summary with the source document through adaptive granularity expansion. These atomic facts represent a more fine-grained unit of information, facilitating detailed understanding and interpretability of the summary's factual inconsistency. Experimental results demonstrate that our proposed factual consistency checking system significantly outperforms existing systems.

FIZZ: Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Document

TL;DR

Experimental results demonstrate that the proposed factual consistency checking system significantly outperforms existing systems and is based on fine-grained atomic facts decomposition.

Abstract

Through the advent of pre-trained language models, there have been notable advancements in abstractive summarization systems. Simultaneously, a considerable number of novel methods for evaluating factual consistency in abstractive summarization systems has been developed. But these evaluation approaches incorporate substantial limitations, especially on refinement and interpretability. In this work, we propose highly effective and interpretable factual inconsistency detection method metric Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Document for abstractive summarization systems that is based on fine-grained atomic facts decomposition. Moreover, we align atomic facts decomposed from the summary with the source document through adaptive granularity expansion. These atomic facts represent a more fine-grained unit of information, facilitating detailed understanding and interpretability of the summary's factual inconsistency. Experimental results demonstrate that our proposed factual consistency checking system significantly outperforms existing systems.
Paper Structure (31 sections, 5 equations, 4 figures, 11 tables, 2 algorithms)

This paper contains 31 sections, 5 equations, 4 figures, 11 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Summarization Factual Consistency Evaluation
  4. Atomic Facts Generation
  5. FIZZ
  6. Coreference Resolution
  7. Atomic Facts Decomposition
  8. Atomic Facts Generation
  9. Atomic Facts Filtering
  10. Atomic Facts Scoring
  11. Atomic Facts Pair-Wise Scoring
  12. Adaptive Granularity Expansion
  13. Experiments
  14. Experimental Setups
  15. Benchmark Datasets
  16. ...and 16 more sections

Figures (4)

  • Figure 1: Comparison between sentence level evaluation and atomic facts level evaluation. The numbers in parentheses represent the maximum NLI entailment scores obtained by comparing each sentence and atomic fact with the source document on a sentence-wise basis.
  • Figure 2: Overall flow of FIZZ. The pipeline begins by applying coreference resolution to both the summary and the document. Atomic facts are then decomposed from the summary using an LLM. These atomic facts are filtered and subsequently scored against the document. The scores are refined through granularity expansion. The ultimate score is defined by choosing the minimum score.
  • Figure 3: The effect of granularity expansions and coreference resolution in real AggreFact dataset. The entailment score of an atomic fact and document sentence with (a) only Coreference Resolution, (b) only Granularity Expansion, and (c) the both.
  • Figure 4: Drawbacks of atomic fact level evaluation versus the sentence level evaluation. The numbers represent the maximum NLI entailment scores obtained by comparing each sentence and atomic fact with the source document on a sentence-wise basis.