A Single Model Ensemble Framework for Neural Machine Translation using Pivot Translation
Seokjin Oh, Keonwoong Noh, Woohwan Jung
TL;DR
The paper addresses translation for low-resource language pairs and black-box models by proposing PivotE, a pivot-based single-model ensemble that first generates diverse translation candidates via multiple pivot paths using one multilingual MT model, and then aggregates the best candidates through quality estimation and a merging module. The approach leverages knowledge transfer from high-resource pivot languages to improve candidate quality and diversity, while avoiding the computational burden of training and ensembling multiple models. Empirical results across distant and similar language pairs show that PivotE consistently outperforms strong baselines, including multi-model ensembles, with substantial reductions in computational overhead. The analysis includes ablations on the number of top candidates, pivot-resource levels, and merging strategies, as well as qualitative case studies demonstrating improved handling of context, homonyms, and domain-specific terminology.
Abstract
Despite the significant advances in neural machine translation, performance remains subpar for low-resource language pairs. Ensembling multiple systems is a widely adopted technique to enhance performance, often accomplished by combining probability distributions. However, the previous approaches face the challenge of high computational costs for training multiple models. Furthermore, for black-box models, averaging token-level probabilities at each decoding step is not feasible. To address the problems of multi-model ensemble methods, we present a pivot-based single model ensemble. The proposed strategy consists of two steps: pivot-based candidate generation and post-hoc aggregation. In the first step, we generate candidates through pivot translation. This can be achieved with only a single model and facilitates knowledge transfer from high-resource pivot languages, resulting in candidates that are not only diverse but also more accurate. Next, in the aggregation step, we select k high-quality candidates from the generated candidates and merge them to generate a final translation that outperforms the existing candidates. Our experimental results show that our method produces translations of superior quality by leveraging candidates from pivot translation to capture the subtle nuances of the source sentence.