SinLlama -- A Large Language Model for Sinhala

TL;DR

The paper tackles the limited Sinhala support in open-source decoder LLMs by extending Llama-3-8B with Sinhala-specific vocabulary and continual pretraining on a 10.7M-sentence corpus to produce SinLlama. It then achieves task-specific competence by fine-tuning SinLlama with three Sinhala text classification datasets using carefully chosen prompts and LoRA, surpassing Llama baselines by a large margin. This work demonstrates the viability of targeted tokenizer adaptation and continual pretraining for low-resource languages and provides an openly released Sinhala decoder LLM for downstream tasks. Overall, the study offers a practical pathway to reduce the digital divide for Sinhala-speaking communities through accessible, open-source technology.

Abstract

Low-resource languages such as Sinhala are often overlooked by open-source Large Language Models (LLMs). In this research, we extend an existing multilingual LLM (Llama-3-8B) to better serve Sinhala. We enhance the LLM tokenizer with Sinhala specific vocabulary and perform continual pre-training on a cleaned 10 million Sinhala corpus, resulting in the SinLlama model. This is the very first decoder-based open-source LLM with explicit Sinhala support. When SinLlama was instruction fine-tuned for three text classification tasks, it outperformed base and instruct variants of Llama-3-8B by a significant margin.

Figures (2)

• Figure 1: Prompt template for supervised fine-tuning
• Figure 2: An example experimental setup for the Writing Style Classification dataset.