Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Data

TL;DR

This work tackles scaling multilingual adaptation of large language models by introducing a massive bilingual corpus (MaLA) and four EMMA-500 Llama 3/3.1 CPT models trained with both monolingual and bilingual data. Through extensive evaluation across 12 benchmarks and 7 tasks, the study shows that bilingual parallel data generally enhances cross-lingual transfer and translation, especially for low-resource languages, while larger pre-trained bases pose adaptation challenges. EMMA-500 achieves state-of-the-art performance on Flores-200 translation and remains competitive on classification and reasoning tasks, underscoring the value of parallel data for multilingual robustness. The work also provides open resources and discusses limitations, data mix design, and the need for broader native-language benchmarks and safety evaluations before real-world deployment.

Abstract

This paper investigates a critical design decision in the practice of massively multilingual continual pre-training -- the inclusion of parallel data. Specifically, we study the impact of bilingual translation data for massively multilingual language adaptation of the Llama3 family of models to 500 languages. To this end, we construct the MaLA bilingual translation corpus, containing data from more than 2,500 language pairs. Subsequently, we develop the EMMA-500 Llama 3 suite of four massively multilingual models -- continually pre-trained from the Llama 3 family of base models extensively on diverse data mixes up to 671B tokens -- and explore the effect of continual pre-training with or without bilingual translation data. Comprehensive evaluation across 7 tasks and 12 benchmarks demonstrates that bilingual data tends to enhance language transfer and performance, particularly for low-resource languages. We open-source the MaLA corpus, EMMA-500 Llama 3 suite artefacts, code, and model generations.

Figures (5)

• Figure 1: Composition by language resource levels of different data types.
• Figure 2: Two date mixes and their composition. The bilingual mix includes all types of data. The monolingual mix consists of a subset of the bilingual mix that excludes bilingual data.
• Figure 3: Comparison of monolingual and bilingual CPT. The scores are averaged across all evaluated languages of the corresponding benchmarks. The baseline model LlaMAX does not have a CPT variant trained on Llama 3.1. Our models show a tendency for bilingual CPT to be better than monolingual CPT in most benchmarks and a remarkable advance on the Flores200 translation benchmark.
• Figure 4: Model adaptability measured by the number of benchmarks on which CPT models are worse than the base model. CPT on LLaMA 2 (L2 Mono) shows a negligible BERTScore drop on MassiveSumm. More highly optimized models such as LLaMA 3 and 3.1 present greater challenges for effective continual pre-training compared to LLaMA 2, especially for high-resource languages, while the situation slightly eases for low-resource languages.
• Figure 5: Numbers of segments and tokens across all language pairs in the MaLA bilingual translation corpus.