Towards a More Inclusive AI: Progress and Perspectives in Large Language Model Training for the Sámi Language

TL;DR

This paper tackles the challenge of training large language models for Ultra Low Resource Languages by focusing on Northern Sámi and compiling the SALT dataset (~22 million tokens). It systematically compares decoder-only BLOOM and sequence-to-sequence Pegasus under mono- and multilingual pretraining (including Finnish and Norwegian) on 8×A100 GPUs, finding that decoder-only models with sequential multilingual training and semantically related pretraining perform best. The study highlights the limited cross-lingual transfer for ULRLs, the advantages of semantically related pretraining, and the general benefit of multilingual training while noting its dependence on semantic overlap. The SALT dataset and code are released to accelerate future research and enable broader NLP applications for Sámi communities, supporting more inclusive language technology development.

Abstract

Sámi, an indigenous language group comprising multiple languages, faces digital marginalization due to the limited availability of data and sophisticated language models designed for its linguistic intricacies. This work focuses on increasing technological participation for the Sámi language. We draw the attention of the ML community towards the language modeling problem of Ultra Low Resource (ULR) languages. ULR languages are those for which the amount of available textual resources is very low, and the speaker count for them is also very low. ULRLs are also not supported by mainstream Large Language Models (LLMs) like ChatGPT, due to which gathering artificial training data for them becomes even more challenging. Mainstream AI foundational model development has given less attention to this category of languages. Generally, these languages have very few speakers, making it hard to find them. However, it is important to develop foundational models for these ULR languages to promote inclusion and the tangible abilities and impact of LLMs. To this end, we have compiled the available Sámi language resources from the web to create a clean dataset for training language models. In order to study the behavior of modern LLM models with ULR languages (Sámi), we have experimented with different kinds of LLMs, mainly at the order of $\sim$ seven billion parameters. We have also explored the effect of multilingual LLM training for ULRLs. We found that the decoder-only models under a sequential multilingual training scenario perform better than joint multilingual training, whereas multilingual training with high semantic overlap, in general, performs better than training from scratch.This is the first study on the Sámi language for adapting non-statistical language models that use the latest developments in the field of natural language processing (NLP).

Figures (2)

• Figure 1: Comparison of translations by ChatGPT and Neurotolge.
• Figure 2: The distribution of tokens and sentences across different domains in the SALT dataset is shown.