Less is More for Improving Automatic Evaluation of Factual Consistency

TL;DR

This work targets automatic evaluation of factual consistency in generated text, identifying noise and robustness weaknesses in the prior AlignScore approach. It introduces LIM-RA, a DeBERTa-based model trained on a cleaned, smaller training set (about 10% of AlignScore) plus synthetic robustness data to enhance resilience to name and number perturbations. Across 33 datasets spanning four benchmarks, LIM-RA achieves state-of-the-art performance, often surpassing GPT-based baselines, while maintaining competitive inference times. The method demonstrates data efficiency and robustness, making it well-suited for evaluating factuality in both traditional NLG outputs and large language model responses, with practical implications for safer and more reliable NLG systems.

Abstract

Assessing the factual consistency of automatically generated texts in relation to source context is crucial for developing reliable natural language generation applications. Recent literature proposes AlignScore which uses a unified alignment model to evaluate factual consistency and substantially outperforms previous methods across many benchmark tasks. In this paper, we take a closer look of datasets used in AlignScore and uncover an unexpected finding: utilizing a smaller number of data points can actually improve performance. We process the original AlignScore training dataset to remove noise, augment with robustness-enhanced samples, and utilize a subset comprising 10\% of the data to train an improved factual consistency evaluation model, we call LIM-RA (Less Is More for Robust AlignScore). LIM-RA demonstrates superior performance, consistently outperforming AlignScore and other strong baselines like ChatGPT across four benchmarks (two utilizing traditional natural language generation datasets and two focused on large language model outputs). Our experiments show that LIM-RA achieves the highest score on 24 of the 33 test datasets, while staying competitive on the rest, establishing the new state-of-the-art benchmarks.

Figures (4)

• Figure 1: Ablation study on using the first $n$ samples from each sub-train dataset for training and overall model performance. We see that the optimum benchmark performance is 452K and 1.6M samples for DeBERTa and RoBERTa respectively. For comparison AlignScore uses 4.7M or the first 500K. Performance broken down by benchmark can be found in \ref{['sec:appendix_ablation']}
• Figure 2: Overall workflow of our method is depicted in the diagram. The top workflow describes how we clean the training data, the bottom workflow illustrates the process of creating synthetic robustness data. Then we train a pre-trained DeBERTa model on those data to obtain LIM-RA.
• Figure 3: Visual description of AlignScore. The context and claim are split into 350 token and sentence chunks respectively. Then an alignment function evaluates each (context chunk, claim sentence). The factual consistency score is calculated by first selecting the highest alignment score for each claim and then averaging these scores across all claims.
• Figure 4: Ablation study on using the first $n$ samples for training and model performance on each benchmark data set.