LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languages

TL;DR

This work addresses the translation performance gaps of large language models for low-resource languages by performing massive multilingual continual pre-training on LLaMA2, yielding LLaMAX that supports over 100 languages. It systematically analyzes vocabulary design and data augmentation, finding that preserving the original vocabulary and leveraging parallel data for augmentation provides the best balance between multilingual coverage and performance. Through large-scale training and subsequent instruction-tuning, LLaMAX achieves substantial gains in Flores-101 and Flores-200 benchmarks, closing the gap with specialized translation systems while maintaining general capabilities and mitigating catastrophic forgetting. The approach offers a robust multilingual foundation model with open-source resources for broad multilingual NLP tasks and downstream fine-tuning.

Abstract

Large Language Models (LLMs) demonstrate remarkable translation capabilities in high-resource language tasks, yet their performance in low-resource languages is hindered by insufficient multilingual data during pre-training. To address this, we conduct extensive multilingual continual pre-training on the LLaMA series models, enabling translation support across more than 100 languages. Through a comprehensive analysis of training strategies, such as vocabulary expansion and data augmentation, we develop LLaMAX. Remarkably, without sacrificing its generalization ability, LLaMAX achieves significantly higher translation performance compared to existing open-source LLMs (by more than 10 spBLEU points) and performs on-par with specialized translation model (M2M-100-12B) on the Flores-101 benchmark. Extensive experiments indicate that LLaMAX can serve as a robust multilingual foundation model. The code \footnote{\url{https://github.com/CONE-MT/LLaMAX/.}} and the models \footnote{\url{https://huggingface.co/LLaMAX/.}} are publicly available.

Figures (7)

• Figure 1: We assess translations in both directions, X$\rightarrow$LG and LG$\rightarrow$X, across various models using Flores-101 test, with X representing all 101 languages included in Flores-101. The results are visualized in a figure where different markers represent various models, a red marker indicates that the language (LG) is Arabic, while a blue marker indicates English. We count the number of translation directions that achieve a spBLEU score higher than 10. The findings indicate that modest LLMs demonstrate strong support for English-centric translation, but underperform in Arabic-centric translation.
• Figure 2: A case illustrating the detailed process of constructing pseudo-parallel data using multilingual dictionary from monolingual or parallel data sources.
• Figure 3: Comparison results between instruction-tuning our multilingual enhanced model and the base model with specialized instruction data. We take X-CSQA, XNLI, MGSM as three examples tasks.
• Figure 4: Comparison results between LLaMAX2-Alpaca and LLaMA2-Alpaca on Flores-200. Some non-English languages are not covered in Flores-200, but LLaMAX2 also boosts its translation performance.
• Figure 5: Correlation between embedding quality and fertility. The embedding quality of LLaMA2 is measured by cosine similarity and Recall@1 on Flores-101 test. Fertility refers to the ratio of the length of a sentence after tokenization compared to its length before tokenization. A high fertility may result in a poor quality of embedding.