Vikhr: The Family of Open-Source Instruction-Tuned Large Language Models for Russian

TL;DR

Vikhr tackles the challenge of building high-quality Russian LLMs by adapting English-oriented models through tokenizer reconstruction, full-parameter continued pre-training with KL regularization to prevent forgetting, and instruction tuning on expanded Russian-English datasets. The pipeline yields Vikhr, an open-source bilingual LLM with a 40k Russian vocabulary, trained on ~11B filtered tokens and evaluated on ruXNLI and Ru-MMLU benchmarks, outperforming the base Mistral setup in Russian generation. The authors demonstrate improved generation quality, computational efficiency, and robust multilingual capabilities, validating a practical workflow for language-specific LLM adaptation. This work provides a reproducible path to extend open-source LLMs to non-English languages and lays groundwork for additional language coverage.

Abstract

There has been a surge in the development of various Large Language Models (LLMs). However, text generation for languages other than English often faces significant challenges, including poor generation quality and reduced computational performance due to the disproportionate representation of tokens in the model's vocabulary. In this work, we address these issues by developing a pipeline for the adaptation of English-oriented pre-trained models to other languages and constructing efficient bilingual LLMs. Using this pipeline, we construct Vikhr, a series of bilingual open-source instruction-following LLMs designed specifically for the Russian language. ``Vikhr'' refers to the name of the Mistral LLM series and means a ``strong gust of wind.'' Unlike previous Russian-language models that typically rely on LoRA adapters on top of English-oriented models, sacrificing performance for lower training costs, Vikhr features an adapted tokenizer vocabulary and undergoes the continued pre-training and instruction tuning of all weights. This not only enhances the model's performance but also significantly improves its computational and contextual efficiency. We also expanded the instruction datasets and corpora for continued pre-training. The model weights, instruction sets, and code are publicly available.

Figures (1)

• Figure 1: The efficiency of the Vikhr tokenizer for Russian in comparison to tokenizers of other models.