Improving Estonian Text Simplification through Pretrained Language Models and Custom Datasets

TL;DR

This work tackles Estonian text simplification under low-resource constraints by comparing an MT-based OpenNMT model with a fine-tuned LLaMA language model, using a newly created Estonian Simplification Dataset assembled from translated Turk/Wiki material and GPT-4.0-generated simplifications. Automatic metrics show a trade-off between BLEU (favoring OpenNMT) and SARI/FKGL (favoring LLaMA), while manual evaluation reveals LLaMA-3.1 more effective at preserving meaning and producing readable, simplified text. The combination of targeted data generation, persona-inspired prompting, and efficient fine-tuning demonstrates the strong potential of LLMs for EstonianATS, with practical implications for accessibility and language technology in low-resource languages. Future work includes dataset expansion, more human corrections, and exploring document-level simplification.

Abstract

This study introduces an approach to Estonian text simplification using two model architectures: a neural machine translation model and a fine-tuned large language model (LLaMA). Given the limited resources for Estonian, we developed a new dataset, the Estonian Simplification Dataset, combining translated data and GPT-4.0-generated simplifications. We benchmarked OpenNMT, a neural machine translation model that frames text simplification as a translation task, and fine-tuned the LLaMA model on our dataset to tailor it specifically for Estonian simplification. Manual evaluations on the test set show that the LLaMA model consistently outperforms OpenNMT in readability, grammaticality, and meaning preservation. These findings underscore the potential of large language models for low-resource languages and provide a basis for further research in Estonian text simplification.

Figures (2)

• Figure 1: Breakdown of sentences simplified using GPT-4.0
• Figure 2: Comparison of Model Performance on BLEU, SARI, and FKGL Metrics for Llama 3.1 and OpenNMT