KyrgyzBERT: A Compact, Efficient Language Model for Kyrgyz NLP

TL;DR

KyrgyzBERT addresses the scarcity of Kyrgyz NLP resources by providing the first monolingual BERT for Kyrgyz, trained from scratch with a custom WordPiece tokenizer on a compact 35.9M-parameter architecture. The authors also create Kyrgyz-SST2, a high-quality sentiment benchmark derived from SST-2 with a gold-standard Kyrgyz test set, enabling reliable evaluation. Finetuning KyrgyzBERT on Kyrgyz-SST2 yields F1=0.8280, competitive with a fivefold larger mBERT, demonstrating strong efficiency and task-specific effectiveness for a low-resource language. The work releases all artifacts publicly, establishing a foundational, accessible NLP toolkit for Kyrgyz and guiding future research and larger-scale pre-training as data grows.

Abstract

Kyrgyz remains a low-resource language with limited foundational NLP tools. To address this gap, we introduce KyrgyzBERT, the first publicly available monolingual BERT-based language model for Kyrgyz. The model has 35.9M parameters and uses a custom tokenizer designed for the language's morphological structure. To evaluate performance, we create kyrgyz-sst2, a sentiment analysis benchmark built by translating the Stanford Sentiment Treebank and manually annotating the full test set. KyrgyzBERT fine-tuned on this dataset achieves an F1-score of 0.8280, competitive with a fine-tuned mBERT model five times larger. All models, data, and code are released to support future research in Kyrgyz NLP.

Figures (1)

• Figure 1: Model performance (F1-Score) vs. size (millions of parameters). KyrgyzBERT achieves performance competitive with mBERT while being 5x smaller, demonstrating superior computational efficiency.