Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SEval-Ex: A Statement-Level Framework for Explainable Summarization Evaluation

Tanguy Herserant, Vincent Guigue

TL;DR

SEval-Ex tackles the challenge of evaluating abstractive summaries with high factual consistency while preserving interpretability. It introduces a statement-level approach that first extracts atomic statements from source and summary and then reasons over their verdicts to compute a consistency score. On SummEval, SEval-Ex achieves state-of-the-art correlation with human consistency judgments (approximately 0.580), surpassing GPT-4-based evaluators, while providing detailed, statement-level evidence. The work demonstrates robustness to hallucinations and discusses practical trade-offs and future extensions toward document-level assessment and efficiency.

Abstract

Evaluating text summarization quality remains a critical challenge in Natural Language Processing. Current approaches face a trade-off between performance and interpretability. We present SEval-Ex, a framework that bridges this gap by decomposing summarization evaluation into atomic statements, enabling both high performance and explainability. SEval-Ex employs a two-stage pipeline: first extracting atomic statements from text source and summary using LLM, then a matching between generated statements. Unlike existing approaches that provide only summary-level scores, our method generates detailed evidence for its decisions through statement-level alignments. Experiments on the SummEval benchmark demonstrate that SEval-Ex achieves state-of-the-art performance with 0.580 correlation on consistency with human consistency judgments, surpassing GPT-4 based evaluators (0.521) while maintaining interpretability. Finally, our framework shows robustness against hallucination.

SEval-Ex: A Statement-Level Framework for Explainable Summarization Evaluation

TL;DR

SEval-Ex tackles the challenge of evaluating abstractive summaries with high factual consistency while preserving interpretability. It introduces a statement-level approach that first extracts atomic statements from source and summary and then reasons over their verdicts to compute a consistency score. On SummEval, SEval-Ex achieves state-of-the-art correlation with human consistency judgments (approximately 0.580), surpassing GPT-4-based evaluators, while providing detailed, statement-level evidence. The work demonstrates robustness to hallucinations and discusses practical trade-offs and future extensions toward document-level assessment and efficiency.

Abstract

Evaluating text summarization quality remains a critical challenge in Natural Language Processing. Current approaches face a trade-off between performance and interpretability. We present SEval-Ex, a framework that bridges this gap by decomposing summarization evaluation into atomic statements, enabling both high performance and explainability. SEval-Ex employs a two-stage pipeline: first extracting atomic statements from text source and summary using LLM, then a matching between generated statements. Unlike existing approaches that provide only summary-level scores, our method generates detailed evidence for its decisions through statement-level alignments. Experiments on the SummEval benchmark demonstrate that SEval-Ex achieves state-of-the-art performance with 0.580 correlation on consistency with human consistency judgments, surpassing GPT-4 based evaluators (0.521) while maintaining interpretability. Finally, our framework shows robustness against hallucination.
Paper Structure (16 sections, 3 equations, 3 figures, 2 tables)

This paper contains 16 sections, 3 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Problem Formalization
  5. Statement Extraction Function:
  6. Statement Verification:
  7. Score Function:
  8. Pipeline Improvements for Long Texts
  9. Experiments
  10. Correlation with Human Judgment
  11. Hallucination Detection Analysis
  12. Hallucination Types
  13. Dataset Preparation
  14. Results Analysis
  15. Discussions and Limitations
  16. ...and 1 more sections

Figures (3)

  • Figure 1: SEval-Ex evaluation pipeline. First, an LLM extract statements during the (1) Statement Extraction phase, then during (2) Verdict Reasoning phase, an LLM labels the statements. Finally, a (3) parser extract the confusion matrix that made the score.
  • Figure 2: Examples of hallucinations divide in 3 types: Entity Replacement, Incorrect Events and Fictitious Details.
  • Figure 3: Comparison of average metric scores across different hallucination types, showing the impact on SEval-Ex score.