Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Comprehensive Benchmark of Language Models on Unicode and Romanized Sinhala

Minuri Rajapakse, Ruvan Weerasinghe

TL;DR

This work presents the first intrinsic, script-aware benchmark of modern foundational LMs for Sinhala, evaluating Unicode and Romanized Sinhala to reveal cross-script performance differences. It combines a bilingual parallel corpus with perplexity assessment for open-source models and qualitative sentence completion for closed-source models, highlighting that script, data composition, and architectural choices drive performance more than size alone. Key findings show Mistral-Nemo-Base-2407 excels on Unicode Sinhala while Mistral-7B-v0.3 shines on Romanized Sinhala, with Llama-3.1-8B offering robust cross-script performance; open-source models benefit from high-quality pretraining data, whereas closed-source models vary by script. The results guide practitioners in selecting or combining models for Sinhala applications and underscore the need for balanced, cross-script training data and future cross-script evaluation enhancements.

Abstract

The performance of Language Models (LMs) on lower-resource, morphologically rich languages like Sinhala remains under-explored, particularly for Romanized Sinhala, which is prevalent in digital communication. This paper presents a comprehensive benchmark of modern LMs on a diverse corpus of Unicode and Romanized Sinhala. We evaluate open-source models using perplexity, a measure of how well a model predicts a text, and leading closed-source models via a qualitative analysis of sentence completion. Our findings reveal that the Mistral-Nemo-Base-2407 model achieves the strongest predictive performance on Unicode text and the Mistral-7B-v0.3 model for Romanized text. The results also highlight the strong all-around performance of the Llama-3.1-8B model for both scripts. Furthermore, a significant performance disparity exists among closed-source models: Gemini-1.5-pro and DeepSeek excel at Unicode generation, whereas Claude-3.5-Sonnet is superior at handling Romanized text. These results provide an essential guide for practitioners selecting models for Sinhala-specific applications and highlight the critical role of training data in handling script variations.

A Comprehensive Benchmark of Language Models on Unicode and Romanized Sinhala

TL;DR

This work presents the first intrinsic, script-aware benchmark of modern foundational LMs for Sinhala, evaluating Unicode and Romanized Sinhala to reveal cross-script performance differences. It combines a bilingual parallel corpus with perplexity assessment for open-source models and qualitative sentence completion for closed-source models, highlighting that script, data composition, and architectural choices drive performance more than size alone. Key findings show Mistral-Nemo-Base-2407 excels on Unicode Sinhala while Mistral-7B-v0.3 shines on Romanized Sinhala, with Llama-3.1-8B offering robust cross-script performance; open-source models benefit from high-quality pretraining data, whereas closed-source models vary by script. The results guide practitioners in selecting or combining models for Sinhala applications and underscore the need for balanced, cross-script training data and future cross-script evaluation enhancements.

Abstract

The performance of Language Models (LMs) on lower-resource, morphologically rich languages like Sinhala remains under-explored, particularly for Romanized Sinhala, which is prevalent in digital communication. This paper presents a comprehensive benchmark of modern LMs on a diverse corpus of Unicode and Romanized Sinhala. We evaluate open-source models using perplexity, a measure of how well a model predicts a text, and leading closed-source models via a qualitative analysis of sentence completion. Our findings reveal that the Mistral-Nemo-Base-2407 model achieves the strongest predictive performance on Unicode text and the Mistral-7B-v0.3 model for Romanized text. The results also highlight the strong all-around performance of the Llama-3.1-8B model for both scripts. Furthermore, a significant performance disparity exists among closed-source models: Gemini-1.5-pro and DeepSeek excel at Unicode generation, whereas Claude-3.5-Sonnet is superior at handling Romanized text. These results provide an essential guide for practitioners selecting models for Sinhala-specific applications and highlight the critical role of training data in handling script variations.
Paper Structure (16 sections, 1 figure, 3 tables)

This paper contains 16 sections, 1 figure, 3 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Benchmarks for Low-Resource and Multilingual Languages
  4. Sinhala-Specific NLP Research
  5. Contribution Within the Research Landscape
  6. METHODOLOGY
  7. Dataset Curation
  8. Models Evaluated
  9. Evaluation Metrics
  10. RESULTS
  11. Perplexity of Open-Source Models
  12. Qualitative Performance of Closed-Source Models
  13. Example Model Outputs
  14. DISCUSSION
  15. Limitations
  16. ...and 1 more sections

Figures (1)

  • Figure 1: PCA plot for the K-Means method