Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Large Language Models for Diacritic Restoration in Romanian Texts: A Comparative Study

Mihai Nadas, Laura Diosan

TL;DR

This study systematically evaluates a broad set of large language models for automatic diacritic restoration in Romanian text, using a progressively designed prompt template strategy and a controlled data corpus derived from dexonline sources. By comparing models from OpenAI, Google, Meta, MistralAI, and RoLlama across zero-shot to three-shot prompts, the authors quantify restoration performance with case-sensitive/insensitive character and word metrics, anchored by an Echo baseline. Key findings show that GPT-4o achieves top performance, and that prompt design (notably three-shot templates) significantly boosts results, while certain open-source models (e.g., Llama variants) exhibit substantial variability and underperformance requiring targeted fine-tuning. The work provides actionable guidance for improving diacritic restoration tools in Romanian and highlights broader implications for prompt engineering and model selection in diacritics-rich languages, with open resources to support reproducibility and further research.

Abstract

Automatic diacritic restoration is crucial for text processing in languages with rich diacritical marks, such as Romanian. This study evaluates the performance of several large language models (LLMs) in restoring diacritics in Romanian texts. Using a comprehensive corpus, we tested models including OpenAI's GPT-3.5, GPT-4, GPT-4o, Google's Gemini 1.0 Pro, Meta's Llama 2 and Llama 3, MistralAI's Mixtral 8x7B Instruct, airoboros 70B, and OpenLLM-Ro's RoLlama 2 7B, under multiple prompt templates ranging from zero-shot to complex multi-shot instructions. Results show that models such as GPT-4o achieve high diacritic restoration accuracy, consistently surpassing a neutral echo baseline, while others, including Meta's Llama family, exhibit wider variability. These findings highlight the impact of model architecture, training data, and prompt design on diacritic restoration performance and outline promising directions for improving NLP tools for diacritic-rich languages.

Evaluating Large Language Models for Diacritic Restoration in Romanian Texts: A Comparative Study

TL;DR

This study systematically evaluates a broad set of large language models for automatic diacritic restoration in Romanian text, using a progressively designed prompt template strategy and a controlled data corpus derived from dexonline sources. By comparing models from OpenAI, Google, Meta, MistralAI, and RoLlama across zero-shot to three-shot prompts, the authors quantify restoration performance with case-sensitive/insensitive character and word metrics, anchored by an Echo baseline. Key findings show that GPT-4o achieves top performance, and that prompt design (notably three-shot templates) significantly boosts results, while certain open-source models (e.g., Llama variants) exhibit substantial variability and underperformance requiring targeted fine-tuning. The work provides actionable guidance for improving diacritic restoration tools in Romanian and highlights broader implications for prompt engineering and model selection in diacritics-rich languages, with open resources to support reproducibility and further research.

Abstract

Automatic diacritic restoration is crucial for text processing in languages with rich diacritical marks, such as Romanian. This study evaluates the performance of several large language models (LLMs) in restoring diacritics in Romanian texts. Using a comprehensive corpus, we tested models including OpenAI's GPT-3.5, GPT-4, GPT-4o, Google's Gemini 1.0 Pro, Meta's Llama 2 and Llama 3, MistralAI's Mixtral 8x7B Instruct, airoboros 70B, and OpenLLM-Ro's RoLlama 2 7B, under multiple prompt templates ranging from zero-shot to complex multi-shot instructions. Results show that models such as GPT-4o achieve high diacritic restoration accuracy, consistently surpassing a neutral echo baseline, while others, including Meta's Llama family, exhibit wider variability. These findings highlight the impact of model architecture, training data, and prompt design on diacritic restoration performance and outline promising directions for improving NLP tools for diacritic-rich languages.

Paper Structure

This paper contains 54 sections, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Research Questions
  3. Background and related work
  4. Diacritics in Natural Language Processing
  5. Existing datasets and characteristics
  6. Diacritics restoration in Romanian text
  7. Methodology
  8. Overall approach
  9. Data Corpus
  10. Selected Data
  11. Selection of Large Language Models
  12. Overview of selected Large Language Models
  13. Prompt template design
  14. Baseline method
  15. Evaluation Metrics
  16. ...and 39 more sections

Figures (3)

  • Figure 1: Performance of Large Language Models (LLMs) in Diacritics Restoration Tasks: Results based on Total Average Score (TAS). The highest score for each model represents the Maximum Total Average Score (MTAS)
  • Figure 2: Reference diacritic distribution (pie chart) and per-model restoration counts (heat-map); darker cells indicate over-generation.
  • Figure 3: Performance of Large Language Models (LLMs) in Diacritics Restoration Tasks: Fully Detailed Results. This table showcases the comparative effectiveness of different LLMs, including OpenAI GPT-4 and its variants, against the Echo model baseline. Highlighted scores demonstrate the LLMs' superior capability in accurately restoring diacritics in Romanian texts across various prompt templates, with detailed insights into the impact of model architecture and prompt complexity on restoration accuracy.