Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resources

TL;DR

This work systematically evaluates 36Continual Pretraining configurations across three multilingual base models to adapt LLMs to 30+ languages using monolingual, bilingual, and code-augmented data. It reveals that bilingual CPT improves multilingual classification but often causes generation language mixing, while code data robustly boosts classification and language modeling yet can degrade translation quality in some cases. Importantly, it shows that language categories based on cross-lingual transfer (altruistic, selfish, stagnant) do not generalize under varying CPT regimes, highlighting complex interactions in multilingual learning. The findings argue for flexible, adaptive CPT frameworks that balance cross-lingual transfer benefits with generation integrity, aiming to reduce language disparities in practical NLP deployments.

Abstract

Large Language Models (LLMs) exhibit significant disparities in performance across languages, primarily benefiting high-resource languages while marginalizing underrepresented ones. Continual Pretraining (CPT) has emerged as a promising approach to address this imbalance, although the relative effectiveness of monolingual, bilingual, and code-augmented data strategies remains unclear. This study systematically evaluates 36 CPT configurations involving three multilingual base models, across 30+ languages categorized as altruistic, selfish, and stagnant, spanning various resource levels. Our findings reveal three major insights: (1) Bilingual CPT improves multilingual classification but often causes language mixing issues during generation. (2) Including programming code data during CPT consistently enhances multilingual classification accuracy, particularly benefiting low-resource languages, but introduces a trade-off by slightly degrading generation quality. (3) Contrary to prior work, we observe substantial deviations from language classifications according to their impact on cross-lingual transfer: Languages classified as altruistic often negatively affect related languages, selfish languages show conditional and configuration-dependent behavior, and stagnant languages demonstrate surprising adaptability under certain CPT conditions. These nuanced interactions emphasize the complexity of multilingual representation learning, underscoring the importance of systematic studies on generalizable language classification to inform future multilingual CPT strategies.

Figures (6)

• Figure 1: FLORES-200 X-Eng BLEU score comparing bilingual and monolingual CPT across high-, mid-, and low-resource languages.
• Figure 2: SIB-200 classification accuracy comparing monolingual and bilingual CPT across high-, mid-, and low-resource languages.
• Figure 3: FLORES-200 X-Eng BLEU score comparing monolingual and monolingual+code CPT across high-, mid-, and low-resource languages.
• Figure 4: SIB-200 classification accuracy comparing monolingual and monolingual+code CPT across high-, mid-, and low-resource languages.
• Figure 5: SIB-200 classification accuracy comparing bilingual and bilingual+code CPT across high-, mid-, and low-resource languages.