Scaling Laws for Multilingual Language Models

TL;DR

The paper addresses multilingual language model scaling by proposing a language-family–level independence hypothesis, enabling a single, compact scaling law that links test loss to model size $N$, data size $D$, and language-family sampling ratios $\bm{p}$. The joint law takes the form $\mathcal{L}_i(N,D, p_i)=(E_i+\frac{A_i}{N^{\alpha_i}}+\frac{B_i}{D^{\beta_i}})\,p_i^{-\gamma_i}$, with $\gamma_i$ invariant to $N$ and $D$, and validates it across 23 languages in 5 families using over 100 models. The authors demonstrate that optimal sampling ratios $\bm{p}^*$ derived from small models (e.g., 85M) generalize to large models (up to 1.2B), enabling resource-efficient data-mixing strategies for multilingual pretraining. They compare normalization schemes for losses and show that normalized losses yield better balance across families by emphasizing high-$\gamma_i$ groups. Overall, the work provides a scalable framework for predicting multilingual LM performance and guiding data allocation without exhaustively training large models.

Abstract

We propose a novel scaling law for general-purpose decoder-only language models (LMs) trained on multilingual data, tackling the problem of balancing languages during multilingual pretraining. A primary challenge in studying multilingual scaling is the difficulty of analyzing individual language performance due to cross-lingual transfer. To address this, we shift the focus from individual languages to language families. We introduce and validate a hypothesis that the test cross-entropy loss for each language family is determined solely by its own sampling ratio, independent of other languages in the mixture. This insight simplifies the complexity of multilingual scaling and make the analysis scalable to an arbitrary number of languages. Building on this hypothesis, we derive a power-law relationship that links performance with dataset size, model size and sampling ratios. This relationship enables us to predict performance across various combinations of the above three quantities, and derive the optimal sampling ratios at different model scales. To demonstrate the effectiveness and accuracy of our proposed scaling law, we perform a large-scale empirical study, training more than 100 models on 23 languages spanning 5 language families. Our experiments show that the optimal sampling ratios derived from small models (85M parameters) generalize effectively to models that are several orders of magnitude larger (1.2B parameters), offering a resource-efficient approach for multilingual LM training at scale.

Figures (13)

• Figure 1: We propose a multilingual scaling law connecting the test cross-entropy loss ($\mathcal{L}$) with model size in number of parameters ($N$), dataset size ($D$) and sampling ratios for different language families (${\bm{p}}$). The plots illustrate a power-law relationship by varying one quantity while fixing the other two for five language families.
• Figure 2: Left: The Romance loss remains stable as the Germanic sampling ratio varies, indicating minimal cross-family transfer and supporting our hypothesis that each language family’s performance is primarily influenced by its own sampling ratio. Right: In contrast, when both groups contain Indic languages, Group 1 loss decreases as Group 2 sampling ratio increases, demonstrating significant cross-group transfer. This underscores the importance of grouping by language families for accurate analysis. The loss values are normalized by the mean loss at $p=0.2$ to align the plot scales. Shaded areas indicate standard deviation.
• Figure 3: Fitting for Germanic and Slavic families with 50B tokens. The high R-squared values indicate an accurate fit of the power-law relationship.
• Figure 4: Left: For a fixed token count $D$, there is a linear relationship between $\log (\mathcal{L}_i)$ and $\log(p)$ for different values of model size $N$. Right: For a fixed model size $N$, there is a linear relationship between $\log (\mathcal{L}_i)$ and $\log(p)$ for different values of dataset size $D$. The parallel lines indicate that the decay rate $\gamma_i$ does not depend on either $N$ or $D$. Both axes are in log-scale.
• Figure 5: Left & Middle: Fitted law on 50B and 100B training tokens, showing that the scaling law well captures the relationship between loss, model size, dataset size and sampling ratios. Right: Predicted vs. actual losses with our scaling law. The fitting uses the top $80\%$ of the loss data (blue points) and then validated on the lower $20\%$ (orange points). The strong alignment between the predicted and actual losses demonstrates the predictive accuracy of the scaling law.