Revisiting Multilingual Data Mixtures in Language Model Pretraining

TL;DR

This study rigorously tests four prevailing beliefs about multilingual data mixtures in pretraining large language models by training $1.1$B and $3$B parameter decoders on corpora spanning up to $400$ languages and $D$ between $100$B and $225$B tokens. It demonstrates that increasing English data does not inherently harm multilingual performance if sufficient multilingual content is present, and that English serves as an effective pivot across language families, with marginal gains from mixing pivots. Curriculum learning does not mitigate negative interference, and the so-called curse of multilinguality is not simply a function of language count but depends on model capacity and data distribution, with natural distributions showing more resilience. The findings yield practical guidance for multilingual pretraining, emphasizing data quality and diversity over language count and suggesting that future work should explore larger models to examine capacity-related effects further.

Abstract

The impact of different multilingual data mixtures in pretraining large language models (LLMs) has been a topic of ongoing debate, often raising concerns about potential trade-offs between language coverage and model performance (i.e., the curse of multilinguality). In this work, we investigate these assumptions by training 1.1B and 3B parameter LLMs on diverse multilingual corpora, varying the number of languages from 25 to 400. Our study challenges common beliefs surrounding multilingual training. First, we find that combining English and multilingual data does not necessarily degrade the in-language performance of either group, provided that languages have a sufficient number of tokens included in the pretraining corpus. Second, we observe that using English as a pivot language (i.e., a high-resource language that serves as a catalyst for multilingual generalization) yields benefits across language families, and contrary to expectations, selecting a pivot language from within a specific family does not consistently improve performance for languages within that family. Lastly, we do not observe a significant "curse of multilinguality" as the number of training languages increases in models at this scale. Our findings suggest that multilingual data, when balanced appropriately, can enhance language model capabilities without compromising performance, even in low-resource settings

Figures (10)

• Figure 1: Validation LM loss for English and weighted average LM loss of non-English languages (Multilingual) across different proportions of English in the pretraining data for $1.1$B models. (a) In a Fixed Total Budget, increasing English data ($\geq$50%) leads to a performance drop in other languages. (b) In a Fixed Multilingual Budget, increasing English data (up to 60%) does not have a negative effect on other languages.
• Figure 2: Aggregated benchmark performance for English and weighted average of non-English (Multilingual) across different proportions of English in the training data for $1.1$B models. The dashed lines represent the random baselines for each language group. (a) In a Fixed Total Budget, increasing English data ($\geq$50%), does not hurt downstream performance on the Multilingual group. (b) In a Fixed Multilingual Budget, we see that increasing English data has a negligible impact on the Multilingual group's performance.
• Figure 3: Weighted average of validation LM loss for (a) Slavic and (b) Cyrillic-script languages when we have English, Russian, or English+Russian as a pivot language in the training data mix. Having a combination of Russian and English as pivots leads to the best performance for both groups of languages (Model size = $1.1$B).
• Figure 4: LM loss on the validation set for $3$B models as a function of consumed training tokens, shown separately for (a) English, (b) non-English, and (c) pivot languages under different curriculum strategies.
• Figure 5: Average validation LM loss for different language groups ($x$-axis) across various curse of multilinguality experiments that include more languages in the pretraining mixture ($y$-axis). Increasing the number of languages does not necessarily degrade the performance of languages included in previous experiments, provided that the amount of training data (in tokens) for those languages remains the same. (English is excluded from these evaluations)