German Text Simplification: Finetuning Large Language Models with Semi-Synthetic Data

TL;DR

The paper tackles data scarcity in German document-level text simplification by building a semi-synthetic parallel corpus: crawling professionally simplified German texts and using GPT-4 to generate corresponding source texts, which are then used to finetune decoder-only German LLMs up to 13B. It systematically analyzes decoding strategies and shows that larger models improve automatic metrics, while highlighting the misalignment between those metrics and human judgments on real-world data. The study demonstrates that models trained on semi-synthetic data can meaningfully reduce text complexity on real content, and it critically discusses the limitations of rule-based evaluation metrics for this task. By releasing the dataset and models, the work provides a practical, scalable pathway for improving German text simplification and sets directions for more nuanced evaluation and domain-adaptive styling.

Abstract

This study pioneers the use of synthetically generated data for training generative models in document-level text simplification of German texts. We demonstrate the effectiveness of our approach with real-world online texts. Addressing the challenge of data scarcity in language simplification, we crawled professionally simplified German texts and synthesized a corpus using GPT-4. We finetune Large Language Models with up to 13 billion parameters on this data and evaluate their performance. This paper employs various methodologies for evaluation and demonstrates the limitations of currently used rule-based metrics. Both automatic and manual evaluations reveal that our models can significantly simplify real-world online texts, indicating the potential of synthetic data in improving text simplification.

Figures (5)

• Figure 1: Illustration of synthetic data generation. Data is crawled from websites specializing in language simplification. GPT-4 generates texts in everyday language, ensuring the original content remains unaltered. We construct a simplification dataset where these texts serve as input while the crawled simplifications act as reference simplifications.
• Figure 2: In this example, a reference simplification and a model prediction, translated into English, are contextually similar but lack any shared tetra-grams, yielding a BLEU score of zero.
• Figure 3: N-gram precisions for predictions of the leo-hessianai-7b model on the complete test set. We observed significantly sloping precision scores for increasing n-gram sizes
• Figure 4: Example to illustrate a high fragmentation penalty due to varied placement of 'climate crisis', negatively impacting the METEOR Score.
• Figure 5: N-gram deletion precisions for predictions of the leo-hessianai-7b model on the complete test set. The median values of our observed SARI delete precision scores reach high values, especially for tri- and tetra-grams