How Multilingual Are Large Language Models Fine-Tuned for Translation?

TL;DR

The paper investigates whether fine-tuning large language models for translation enables truly massively multilingual MT beyond the supervised language pairs. It performs a comprehensive evaluation of the Tower family on 132 translation tasks across 12 languages, comparing against the NLLB baseline using Flores-200 data and COMET-22 as the primary metric. Results show that translation-focused fine-tuning improves MT ability beyond the trained pairs, including zero-shot languages, but with substantial variance and notable failures for languages like Icelandic and Korean, as well as off-target translations in several cases. The findings suggest promising potential for multilingual MT with LLMs but also call for advances in instruction tuning, model selection strategies, and tokenization fairness to achieve robust, widely transferable performance.

Abstract

A new paradigm for machine translation has recently emerged: fine-tuning large language models (LLM) on parallel text has been shown to outperform dedicated translation systems trained in a supervised fashion on much larger amounts of parallel data (Xu et al., 2024a; Alves et al., 2024). However, it remains unclear whether this paradigm can enable massively multilingual machine translation or whether it requires fine-tuning dedicated models for a small number of language pairs. How does translation fine-tuning impact the MT capabilities of LLMs for zero-shot languages, zero-shot language pairs, and translation tasks that do not involve English? To address these questions, we conduct an extensive empirical evaluation of the translation quality of the TOWER family of language models (Alves et al., 2024) on 132 translation tasks from the multi-parallel FLORES-200 data. We find that translation fine-tuning improves translation quality even for zero-shot languages on average, but that the impact is uneven depending on the language pairs involved. These results call for further research to effectively enable massively multilingual translation with LLMs.

Figures (21)

• Figure 1: Distribution of Comet scores for each translation approach over language pair supervision type: TowerInstruct-13B models are competitive with Nllb on average across settings, suggesting that transfer learning benefits many unsupervised languages, but not all based on the increased variance in the least supervised conditions.
• Figure 2: Spread of translation quality as measured by Comet score for each target language for the dedicated Nllb model (top) and the overall top-performing TowerInstruct-13B (bottom). For each data point, the color identifies the source language, and we mark the best/worst dissimilar paths (D+/D-), the similar path (S), and translation from English (E).
• Figure 3: Models that achieve the absolute best and worst scores per language pair. The group outlined in dotted-blue corresponds to the fully supervised set, the group outlined in dot-dash-red corresponds to the unsupervised set and the remaining pairs are partially supervised. The circle and cross symbols denote whether the language pair consists of similar or dissimilar language, respectively (as defined in Table \ref{['lang_table']}). Symbols marked with asterisks denote statistical significance according to a paired $t$-test at a significance level of 0.05.
• Figure 4: Ratio of on-target translation averaged over the source sentences from the supervised set. The TowerInstruct models are generally on target for supervised languages, but off-target translations are frequent for unsupervised languages. By contrast, Nllb translations are on target for 97-100% of outputs.
• Figure 5: Average input length per language after subword segmentation for each model (top) and average translation quality out of each source language per model (bottom).