Beyond English: Toward Inclusive and Scalable Multilingual Machine Translation with LLMs

TL;DR

This work tackles the English-centric bias in multilingual MT by introducing LMT, a Chinese-English-centric family of models spanning 60 languages and 234 translation directions. It combines Continued Pre-training (CPT) with Supervised Fine-Tuning (SFT) on a rigorously curated, multi-stage data pipeline and adds Parallel Multilingual Prompting (PMP) to enhance cross-lingual transfer, while identifying and mitigating directional degeneration through Strategic Downsampling. The approach achieves state-of-the-art performance for models with similar language coverage, notably with LMT-60-4B outperforming larger baselines like Aya-101-13B and NLLB-54B, and demonstrates strong ablations showing the value of CPT, SD, and PMP. Releasing four model sizes, this work provides a practical, scalable baseline and methodological toolkit for inclusive, high-quality MMT across diverse languages and directions.

Abstract

Large language models have significantly advanced Multilingual Machine Translation (MMT), yet the broad language coverage, consistent translation quality, and English-centric bias remain open challenges. To address these challenges, we introduce \textbf{LMT}, a suite of \textbf{L}arge-scale \textbf{M}ultilingual \textbf{T}ranslation models centered on both Chinese and English, covering 60 languages and 234 translation directions. During development, we identify a previously overlooked phenomenon of \textbf{directional degeneration}, where symmetric multi-way fine-tuning data overemphasize reverse directions (X $\to$ En/Zh), leading to excessive many-to-one mappings and degraded translation quality. We propose \textbf{Strategic Downsampling}, a simple yet effective method to mitigate this degeneration. In addition, we design \textbf{Parallel Multilingual Prompting (PMP)}, which leverages typologically related auxiliary languages to enhance cross-lingual transfer. Through rigorous data curation and refined adaptation strategies, LMT achieves SOTA performance among models of comparable language coverage, with our 4B model (LMT-60-4B) surpassing the much larger Aya-101-13B and NLLB-54B models by a substantial margin. We release LMT in four sizes (0.6B/1.7B/4B/8B) to catalyze future research and provide strong baselines for inclusive, scalable, and high-quality MMT \footnote{\href{https://github.com/NiuTrans/LMT}{https://github.com/NiuTrans/LMT}}.

Figures (11)

• Figure 1: Top: Performance of base LLMs (orange) on the Belebele benchmark across 108 languages, plotted against their data ratios in the CulturaX (blue). Bottom: Bilingual data volume (million sentence pairs) from the OPUS corpus for 60 languages in our study, covering English-centric (blue) and Chinese-centric (orange) directions. Languages are grouped into high-, medium-, and low-resource tiers.
• Figure 2: An overview of our methodology for LMT. The pipeline consists of two main stages: a hybrid data curation process (top) to build the training corpus, and a two-stage adaptation (bottom) involving CPT and SFT.
• Figure 3: Examples of the three prompt formats for the CPT and SFT stages of LMT adaptation. The underlined text indicates the part used for loss computation during training.
• Figure 4: The impact of the Strategic Downsampling proportion ($p$). Dashed lines represent the use of separate, non-symmetric data for the X$\to$En/Zh directions.
• Figure 5: Ablation study on the impact of each component: Strategic Downsampling (SD), Continued Pre-training (CPT), and Parallel Multilingual Prompting (PMP). The annotated values quantify the performance gain of each component over the preceding one.