Gamayun's Path to Multilingual Mastery: Cost-Efficient Training of a 1.5B-Parameter LLM

TL;DR

Gamayun demonstrates that a 1.5B multilingual LLM trained from scratch on 2.5T tokens can achieve strong multilingual performance and state-of-the-art results in Russian among models of similar size, without relying on distillation from larger models. The authors introduce a two-stage pre-training regime that first aligns languages with a balanced multilingual mix and then shifts toward high-quality English data to transfer knowledge across languages. Post-training employs supervised fine-tuning and Direct Preference Optimization with diverse synthetic, instruction, and math data to bolster reasoning and instruction-following, while separately enhancing Russian cultural and linguistic capabilities. The work shows cost-efficient pathways to robust multilingual NLP, offering practical implications for deploying capable language models in resource-constrained settings and for domains requiring strong cross-lingual transfer and cultural knowledge.

Abstract

We present Gamayun, a 1.5B-parameter multilingual language model trained entirely from scratch on 2.5T tokens. Designed for efficiency and deployment in resource-constrained environments, Gamayun addresses the lack of research on small non-English-centric LLMs by adopting a novel two-stage pre-training strategy: balanced multilingual training for cross-lingual alignment, followed by high-quality English enrichment to transfer performance gains across languages. Our model supports 12 languages, with special focus on Russian. Despite a significantly smaller training budget than comparable models, Gamayun outperforms LLaMA3.2-1B (9T tokens) on all considered benchmarks, and surpasses Qwen2.5-1.5B (18T tokens) on a wide range of English and multilingual tasks. It matches or exceeds Qwen3 (36T tokens) on most tasks outside advanced STEM, achieving state-of-the-art results in Russian, including the MERA benchmark, among the models of comparable size (1-2B parameters).

Figures (4)

• Figure 1: First stage (left) and second stage (right) data distributions. In the first stage, non-English data prevails. In contrast, the second stage becomes more English-centric; the amount of lower quality mC4 corpus decreased from 64% to 16%.
• Figure 2: Two-stage training of the 1.5B model: a balanced language mix in the first stage (to the left of the red point) ensures language alignment, which makes the increased proportion of high-quality English data in the second stage beneficial for all languages. We observe that the two-stage model (blue) outperforms the model trained without alignment stage (orange) in average accuracy on both English (top) and multilingual (bottom) benchmarks.
• Figure 3: Loss curve on English dev-set. The orange line is related to single-language model, the red was trained with several languages. Green lines show were the models consumed the same amount of English data.
• Figure 4: Final dev loss of the models trained with different proportions of Russian(yellow) and Chinese (orange) with respect to English. We can see that adding more Chinese tokens can have minimal effect to Chinese loss, while the English loss still degrades. For Russian, the Pareto front is more clear.