ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilinguality

TL;DR

This paper tackles multilingual scaling laws by introducing the Adaptive Transfer Scaling Law (ATLAS), a data-exposure model that accounts for monolingual data, cross-lingual transfer, and data repetition across languages. Using 774 pretraining experiments on MADLAD-400 with 10M–8B parameter models across 50 languages and 48 evaluation languages, the authors derive a 38×38 cross-lingual transfer matrix and a language-agnostic scaling law, enabling robust extrapolation to unseen languages and mixtures. ATLAS outperforms prior scaling laws in both monolingual and multilingual settings, with strong evidence for cross-lingual transfer effects and a quantifiable compute-optimal strategy when expanding language coverage via iso-loss frontiers. The work also provides practical guidance on when to pretrain from scratch versus finetune from multilingual checkpoints and characterizes the curse of multilinguality through a capacity/data scaling framework, offering actionable insights for democratizing scaling laws across languages and scripts.

Abstract

Scaling laws research has focused overwhelmingly on English -- yet the most prominent AI models explicitly serve billions of international users. In this work, we undertake the largest multilingual scaling laws study to date, totaling 774 multilingual training experiments, spanning 10M-8B model parameters, 400+ training languages and 48 evaluation languages. We introduce the Adaptive Transfer Scaling Law (ATLAS) for both monolingual and multilingual pretraining, which outperforms existing scaling laws' out-of-sample generalization often by more than 0.3 R^2. Our analyses of the experiments shed light on multilingual learning dynamics, transfer properties between languages, and the curse of multilinguality. First, we derive a cross-lingual transfer matrix, empirically measuring mutual benefit scores between 38 x 38=1444 language pairs. Second, we derive a language-agnostic scaling law that reveals how to optimally scale model size and data when adding languages without sacrificing performance. Third, we identify the computational crossover points for when to pretrain from scratch versus finetune from multilingual checkpoints. We hope these findings provide the scientific foundation for democratizing scaling laws across languages, and enable practitioners to efficiently scale models -- beyond English-first AI.

Figures (9)

• Figure 1: Optimal Scaling Trajectories for English, French, Russian, Chinese, Hindi, and Swahili. The law for [monolingual vocabulary, monolingual training]=(---), the law for [multilingual vocabulary, monolingual training]=(- - -), and the law for [multilingual vocabulary, unimax training]=(···). We find (1) per-language optimal scaling trajectories are similar, (2) there is a compute efficiency tax for training with multilingual vocabularies or training sets (especially for English), and (3) as Hindi and Swahili observe data repetition their curves slope upward from diminishing returns.
• Figure 2: The Cross-Lingual Transfer Matrix, depicting the measured Language Transfer Score across $30 \times 30$ language pairs. Positive scores indicate more positive transfer, negative scores more interference, during bilingual co-training. The dashed boxes indicate the top-5 source languages for each target language. Refer to Appendix \ref{['app:lang-transfer-bts-estimation']} for full details, and \ref{['fig:language-transfer-heatmap-full']} for the larger $38 \times 38$ matrix. While English is the best source language for many of the languages, we find language similarity is highly predictive of these scores.
• Figure 3: Left: A language transfer scatter plot, comparing score symmetry: is language A as helpful to B as vice versa? Points cluster near the diagonal, indicating strong symmetry. The most synergistic and symmetric pairs almost exclusively share both language family and script. Greater linguistic distance correlates with increased asymmetry and reduced positive transfer. Right: The impact of linguistic similarity (via language family or script) on transfer scores. The box spans the inter-quartile range, with the median line at the center. We find the differences between each group are statistically significant ($p < .001$), suggesting that sharing either a language family or a script independently contribute greater positive cross-lingual transfer.
• Figure 4: We empirically measure the relative degradation in loss (y-axis), as compared to a monolingual model of the same $(N,D)$, from adding pretraining languages (z-axis). Left: We fix $D=25B$ tokens, and vary the model size $N$. Right: We fix $N=2B$ parameters, and vary the tokens $D$. The points are real empirical observations, averaged across languages. The mesh-grid is a surface estimate, using a cubic spline. Target language loss is most affected by the number of training languages, but this loss penalty declines for larger models with more capacity.
• Figure 5: Iso-loss from expanding language coverage: When scaling the number of languages a model serves from $K \rightarrow r \cdot K$, we can estimate how much they need to increase model size $N'/N$ and/or training tokens $D'_{tot}/D_{tot}$, without degrading any of their languages' loss. We derive the compute optimal scaling equations. The iso-losses indicate that a practitioner should expand their compute budget by $C \cdot r^{0.97}$ to expand their language coverage by $r$.