Profiling German Text Simplification with Interpretable Model-Fingerprints

TL;DR

The paper addresses the need for efficient, reproducible diagnosis of German ATS outputs by introducing the Simplification Profiler, a multidimensional fingerprint toolkit that decomposes simplifications into interpretable properties. It combines semantic-fidelity metrics (COR, COV), linguistic-quality metrics (SIM, LNG, FBR, COH), and simple heuristics to form a comprehensive feature set, enabling a linear classifier to distinguish model configurations with high accuracy (F1 up to $71.9\%$). The methodology uses German Wikipedia as a controlled data source and explores broad (prompt strategies and model sizes) and fine-grained (few-shot prompts) variations to demonstrate diagnostic sensitivity. The results show that the Profiler can reveal distinct trade-offs between readability, content preservation, and grammaticality, providing actionable diagnostics for building adaptive, audience-aware German ATS systems. The work emphasizes openness (open-source toolkit) and outlines limitations and avenues for future work, including human correlation studies and cross-language extensions, to further validate and generalize the fingerprint approach.

Abstract

While Large Language Models (LLMs) produce highly nuanced text simplifications, developers currently lack tools for a holistic, efficient, and reproducible diagnosis of their behavior. This paper introduces the Simplification Profiler, a diagnostic toolkit that generates a multidimensional, interpretable fingerprint of simplified texts. Multiple aggregated simplifications of a model result in a model's fingerprint. This novel evaluation paradigm is particularly vital for languages, where the data scarcity problem is magnified when creating flexible models for diverse target groups rather than a single, fixed simplification style. We propose that measuring a model's unique behavioral signature is more relevant in this context as an alternative to correlating metrics with human preferences. We operationalize this with a practical meta-evaluation of our fingerprints' descriptive power, which bypasses the need for large, human-rated datasets. This test measures if a simple linear classifier can reliably identify various model configurations by their created simplifications, confirming that our metrics are sensitive to a model's specific characteristics. The Profiler can distinguish high-level behavioral variations between prompting strategies and fine-grained changes from prompt engineering, including few-shot examples. Our complete feature set achieves classification F1-scores up to 71.9 %, improving upon simple baselines by over 48 percentage points. The Simplification Profiler thus offers developers a granular, actionable analysis to build more effective and truly adaptive text simplification systems.

Figures (3)

• Figure 1: The fingerprints of the three example simplification strategies (S1, S2, S3). Each distinct profile shape highlights the strengths, weaknesses, and trade-offs of the corresponding simplification, such as conciseness versus content coverage.
• Figure 2: Feature importance heatmap showing the distinct metric patterns for identifying each condition. For example, the Target prompt is strongly characterized by readability (FBR) and content coverage (COV), while the 1B model is primarily identified by its length (LEN).
• Figure 3: Overlaid fingerprints of the four main prompting strategies. The distinct shape of each profile highlights the inherent trade-offs of each strategy, such as the focus on readability versus content coverage.