DETECT: Determining Ease and Textual Clarity of German Text Simplifications

TL;DR

DETECT introduces a German-specific learnable metric for automatic text simplification that directly assesses simplicity, meaning preservation, and fluency. It builds a synthetic supervision pipeline and a refined LLM-based rubric to train a RoBERTa-based predictor, validated against human judgments and traditional ATS metrics. The approach yields higher correlations with human judgments than BLEU, SARI, or BERTScore, particularly for meaning preservation, while also illuminating the benefits and limits of LLM-based evaluation in language accessibility tasks. The work offers practical guidelines for leveraging synthetic supervision and rubric refinement in cross-linguistic evaluation, with potential extensions to broader domains and languages.

Abstract

Current evaluation of German automatic text simplification (ATS) relies on general-purpose metrics such as SARI, BLEU, and BERTScore, which insufficiently capture simplification quality in terms of simplicity, meaning preservation, and fluency. While specialized metrics like LENS have been developed for English, corresponding efforts for German have lagged behind due to the absence of human-annotated corpora. To close this gap, we introduce DETECT, the first German-specific metric that holistically evaluates ATS quality across all three dimensions of simplicity, meaning preservation, and fluency, and is trained entirely on synthetic large language model (LLM) responses. Our approach adapts the LENS framework to German and extends it with (i) a pipeline for generating synthetic quality scores via LLMs, enabling dataset creation without human annotation, and (ii) an LLM-based refinement step for aligning grading criteria with simplification requirements. To the best of our knowledge, we also construct the largest German human evaluation dataset for text simplification to validate our metric directly. Experimental results show that DETECT achieves substantially higher correlations with human judgments than widely used ATS metrics, with particularly strong gains in meaning preservation and fluency. Beyond ATS, our findings highlight both the potential and the limitations of LLMs for automatic evaluation and provide transferable guidelines for general language accessibility tasks.

Figures (9)

• Figure 1: Correlation of standard ATS metrics and DEtect with human judgments across simplicity, meaning preservation, fluency, and the total score. Standard ATS metrics (BLEU, SARI, BERTScore) show limited alignment, while DEtect achieves consistently higher correlations, particularly for meaning preservation and overall quality.
• Figure 2: Pipeline of DEtect. The framework consists of four steps: (1) Data curation. Construction of SimpEvalDE from Austrian Press Agency news sentences aligned across CEFR levels (DEPlaIn-APA and LHA-APA); (2) ATS generation. LLMs produce multiple simplifications for each complex sentence; (3) LLM quality scoring. Open-source LLMs are prompted as judges to assign quality scores for simplicity, meaning preservation, and fluency, with prompts iteratively refined based on expert feedback and aligned with a human-designed grading rubric; (4) Evaluation. DEtect, trained on these scores, is validated against human quality judgments using a simplified Rank & Rate protocol maddela2023lens and compared to standard ATS metrics.
• Figure 3: Distribution of Human-Judge assessments averaged across three human evaluators.
• Figure 5: Histograms of score distributions for i) Distill-Qwen-7B, ii) Distill-Llama-8B, and iii) Zephyr-7B, broken down by scoring criterion. The yellow band indicates the IQR, while vertical ticks show the minimum, median, and maximum values. These values are also listed in the box in the upper left corner of each plot.
• Figure 6: Rating of Human-Judge quality scores.