Investigating Multi-Pivot Ensembling with Massively Multilingual Machine Translation Models

TL;DR

The paper tackles translating low-resource directions in massively multilingual NMT and investigates whether multi-pivot ensembling can help. It compares MultiAvg, which averages pivot distributions, against MaxEns, which chooses the most confident pivot per token, formalized as $s(Y;X_M) = \sum_{i=1}^{|Y|} \log( (1/|M|) \sum_{k=1}^{|M|} p(y_i | y_{<i}, X_{\mu_k}) )$ for MultiAvg and $s(Y;X_M) = \sum_{i=1}^{|Y|} \log \max_{k} p(y_i | y_{<i}, X_{\mu_k})$ for MaxEns. On FLORES-101 with M2M100 and SMaLL100 across 20 directions, MultiAvg improves over direct translation but lags behind the English pivot, while MaxEns reduces hallucinations and achieves translation quality competitive with English pivot, albeit with weaker hallucination mitigation in some cases. The results suggest that the most effective pivot strategy is direction-dependent, motivating future work on per-direction pivot selection and combining multi-pivot ensembling with decoding strategies like source-contrastive decoding. Overall, the study highlights both the potential and the limitations of multi-pivot ensembling in massively multilingual NMT.

Abstract

Massively multilingual machine translation models allow for the translation of a large number of languages with a single model, but have limited performance on low- and very-low-resource translation directions. Pivoting via high-resource languages remains a strong strategy for low-resource directions, and in this paper we revisit ways of pivoting through multiple languages. Previous work has used a simple averaging of probability distributions from multiple paths, but we find that this performs worse than using a single pivot, and exacerbates the hallucination problem because the same hallucinations can be probable across different paths. We also propose MaxEns, a novel combination strategy that makes the output biased towards the most confident predictions, hypothesising that confident predictions are less prone to be hallucinations. We evaluate different strategies on the FLORES benchmark for 20 low-resource language directions, demonstrating that MaxEns improves translation quality for low-resource languages while reducing hallucination in translations, compared to both direct translation and an averaging approach. On average, multi-pivot strategies still lag behind using English as a single pivot language, raising the question of how to identify the best pivoting strategy for a given translation direction.

Figures (1)

• Figure 1: A sample translation of Afrikaans to Asturian by using SMaLL100. Translations of individual pivots are shown in the middle, output translations of MaxEns and MultiAvg on the right. MaxEns method eliminates the hallucination, as it follows the more confident pivot (here, Spanish). Glosses in English are presented within gray boxes.