Assessing the Impact of Typological Features on Multilingual Machine Translation in the Age of Large Language Models

TL;DR

The paper investigates whether typological features can predict translation quality in large multilingual MT systems, addressing the question in the era of large language models. It analyzes two prominent models, NLLB-200 (encoder–decoder) and Tower+ (decoder-only), across FLORES+ with 212 target languages, using continuous typological features and an approximate resourcedness proxy. The findings show target language typology significantly influences NLLB-200 performance, with decoding width interactions suggesting language-specific decoding strategies; Tower+ reveals typology-driven patterns that are more constrained by model coverage and data availability. The authors release a fine-grained typological feature set for 212 FLORES+ languages to support future research on language-specific decoding and cross-lingual MT. Overall, the work highlights the practical importance of incorporating typology into decoding decisions and model evaluation in multilingual MT systems.

Abstract

Despite major advances in multilingual modeling, large quality disparities persist across languages. Besides the obvious impact of uneven training resources, typological properties have also been proposed to determine the intrinsic difficulty of modeling a language. The existing evidence, however, is mostly based on small monolingual language models or bilingual translation models trained from scratch. We expand on this line of work by analyzing two large pre-trained multilingual translation models, NLLB-200 and Tower+, which are state-of-the-art representatives of encoder-decoder and decoder-only machine translation, respectively. Based on a broad set of languages, we find that target language typology drives translation quality of both models, even after controlling for more trivial factors, such as data resourcedness and writing script. Additionally, languages with certain typological properties benefit more from a wider search of the output space, suggesting that such languages could profit from alternative decoding strategies beyond the standard left-to-right beam search. To facilitate further research in this area, we release a set of fine-grained typological properties for 212 languages of the FLORES+ MT evaluation benchmark.

Figures (11)

• Figure 1: Spearman correlations between continuous language properties and NLLB-200 chrF++ translation quality scores (a character n-gram based translation quality metric, cf.§\ref{['methodology:evaluation']}) at beam size $k=5$. Source language is English. Sample sizes (i.e. number of target languages) for each property are indicated next to their respective bars. Correlations significant at $p<0.05$ are marked with *, at $p<0.01$ with **, at $p<0.001$ with ***.
• Figure 2: Relative increase in chrF++ for NLLB-200, translating from English to 124 different target languages. Curves are colored by their optimal beam size.
• Figure 3: Tower+ 9B chrF++ scores vs. NLLB-200 3.3B chrF++ scores at beam size $k=7$. Each point denotes a language pair and is colored by source language, while $\blacktriangledown$ denotes target languages officially supported by Tower+. The blue and orange shaded regions indicate language pairs for which either NLLB-200 or Tower+ scores are higher, respectively. Sample size is $n = 7 \times 52 = 364$.
• Figure 4: NLLB-200 translation performance measured at four beam sizes via translation quality metrics (BLEU, chrF++, COMET) and output sequence generation probabilities. Performance is averaged across 124 target languages for individual source languages: Arabic, English, Italian, Dutch, Turkish, Ukrainian, and Vietnamese.
• Figure 5: Target language head-finality (top) and moving average type-token ratio (bottom) vs. chrF++ scores at beam size 5. Data for translations by NLLB-200 from English into 52 target languages.