Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluation

TL;DR

The paper critically evaluates automatic factuality metrics for summarization and shows that most metrics struggle on harder, reasoning-requiring cases and are susceptible to manipulation via harmless edits; prompting-based evaluations (notably ChatGPT-DA) are more robust but may rely on internal model knowledge rather than the source document. Through benchmarks spanning AggreFact, GenAudit, and related datasets, the authors demonstrate that many metrics inflate scores with benign edits and can be gamed with minimal content changes, raising questions about what these metrics truly measure. They propose a set of practical recommendations, including saliency-aware scoring and benchmarks that capture factuality severity, to push towards more grounded and reliable evaluation in high-stakes domains. Overall, the work calls for cautious interpretation of current factuality metrics and highlights the need for evaluations that emphasize genuine factual alignment with sources over superficial cues or internal priors.

Abstract

Modern LLMs can now produce highly readable abstractive summaries, to the point that traditional automated metrics for evaluating summary quality, such as ROUGE, have saturated. However, LLMs still sometimes introduce inaccuracies into summaries, i.e., information inconsistent with or unsupported by the corresponding source. Measuring the occurrence of these often subtle factual inconsistencies automatically has proved challenging. This in turn has motivated development of metrics intended to measure the factual consistency of generated summaries against sources. But are these approaches measuring what they purport to? Or are they mostly exploiting artifacts? In this work, we stress test a range of automatic factuality metrics, including specialized models and LLM-based prompting methods, to probe what they actually capture. Using a shallow classifier to separate ``easy'' examples for factual evaluation where surface features suffice from ``hard'' cases requiring deeper reasoning, we find that all metrics show substantial performance drops on the latter. Furthermore, some metrics are more sensitive to benign, fact-preserving edits than to factual corrections. Building on this observation, we demonstrate that most automatic factuality metrics can be gamed, i.e., their scores can be artificially inflated by appending innocuous, content-free sentences to summaries. Among the metrics tested, the prompt based ChatGPT-DA approach is the most robust and reliable. However, this comes with a notable caveat: Prompting LLMs to assess factuality may overly rely on their parametric knowledge rather than the provided reference when making judgments. Taken together, our findings call into question the reliability of current factuality metrics and prompt a broader reflection on what these metrics are truly measuring.

Figures (7)

• Figure 1: Many methods have been proposed to automatically evaluate the factual consistency of summaries with respect to inputs. In this work we critically evaluate such approaches, e.g., by measuring their sensitivity to various manipulations, as shown here.
• Figure 2: Summaries are categorized as easy, medium, or hard based on prediction accuracy and confidence from a shallow MLP. While specialized metrics perform best on easy examples, their performance declines on hard cases. UniEval, MiniCheck and ChatGPT-DA show greater robustness in more challenging settings
• Figure 3: Score differences for each metric between the original summary and summaries edited for factual accuracy by humans (shown in green) and other benign edits (shown in blue).
• Figure 4: GPT-based consistency evaluation is influenced by parametric knowledge. Left: The score gap between supported and unsupported summaries narrows sharply when references are counterfactual but summaries are factually accurate (p < 0.001). Right: The rate of cases where unsupported summaries are scored higher than supported ones rises from 0.2% to 3.1% when references contradict GPT’s world knowledge while summaries remain factually correct.
• Figure 5: Pairwise score differences across summaries manipulated using four distinct strategies: Adding constant strings (top phrase and assertion phrase) and appending them to summaries (summ + top, and summ + assertion. The results reveal that NLI and bespoke model metrics are particularly vulnerable to gaming, with significant score inflation observed under these manipulations.