Evaluation Under Imperfect Benchmarks and Ratings: A Case Study in Text Simplification

TL;DR

The paper tackles unreliable evaluation in text simplification by identifying benchmark quality and annotator disagreement as core issues. It introduces SynthSimpliEval, a synthetic benchmark combining human-written and model-generated content to better reflect modern systems, and validates it with high annotator agreement and a strong link between human ratings and model size. It then shows that an LLMs-as-a-Jury can provide consistent auto-evaluation, with unified scoring yielding high inter-judge agreement, and that such judgments better track model-size differences than traditional metrics. Finally, it demonstrates that learnable metrics can benefit from synthetic, LLM-rated data, offering a practical path toward reliable evaluation in settings where high-quality annotations are scarce or noisy.

Abstract

Despite the successes of language models, their evaluation remains a daunting challenge for new and existing tasks. We consider the task of text simplification, commonly used to improve information accessibility, where evaluation faces two major challenges. First, the data in existing benchmarks might not reflect the capabilities of current language models on the task, often containing disfluent, incoherent, or simplistic examples. Second, existing human ratings associated with the benchmarks often contain a high degree of disagreement, resulting in inconsistent ratings; nevertheless, existing metrics still have to show higher correlations with these imperfect ratings. As a result, evaluation for the task is not reliable and does not reflect expected trends (e.g., more powerful models being assigned higher scores). We address these challenges for the task of text simplification through three contributions. First, we introduce SynthSimpliEval, a synthetic benchmark for text simplification featuring simplified sentences generated by models of varying sizes. Through a pilot study, we show that human ratings on our benchmark exhibit high inter-annotator agreement and reflect the expected trend: larger models produce higher-quality simplifications. Second, we show that auto-evaluation with a panel of LLM judges (LLMs-as-a-jury) often suffices to obtain consistent ratings for the evaluation of text simplification. Third, we demonstrate that existing learnable metrics for text simplification benefit from training on our LLMs-as-a-jury-rated synthetic data, closing the gap with pure LLMs-as-a-jury for evaluation. Overall, through our case study on text simplification, we show that a reliable evaluation requires higher quality test data, which could be obtained through synthetic data and LLMs-as-a-jury ratings.

Figures (6)

• Figure 1: Average human ratings of each model size, normalized to 0-1. We see that larger models consistently receive higher scores.
• Figure 2: Temperature ablation on SynthSimpliEval. Spearman rank correlations from left to right are 0.626, 0.622, 0.615, and 0.581 respectively.
• Figure 3: Prompt format ablations on SynthSimpliEval. Spearman rank correlations from left to right are 0.626, 0.335, and 0.648 respectively.
• Figure 4: Spearman correlations on SynthSimpliEval between existing metrics, LLMs, and LLM average with model size. See full correlation matrix in \ref{['appendix:full_correlation_matrix']}.
• Figure 5: Average scores of each simplifier (Llama 3) model. The top row contains previous metrics, and the bottom two rows contain normalized LLM judge scores (§\ref{['sec:evaluating_text_simplification']}).