Multilingual Neural Machine Translation with Knowledge Distillation

TL;DR

Multilingual NMT often sacrifices accuracy when scaling to many language pairs. The authors introduce a knowledge-distillation framework where multiple per-language teacher models guide a single multilingual student, enhanced with selective and top-K distillation and a back-distillation variant. Across IWSLT, WMT, and Ted Talk datasets, the approach closes the gap with per-language models and can even surpass them while using far fewer parameters (e.g., $1/44$ of the total for Ted Talk). These techniques improve generalization and scalability, enabling accurate, efficient multilingual translation across dozens of languages.

Abstract

Multilingual machine translation, which translates multiple languages with a single model, has attracted much attention due to its efficiency of offline training and online serving. However, traditional multilingual translation usually yields inferior accuracy compared with the counterpart using individual models for each language pair, due to language diversity and model capacity limitations. In this paper, we propose a distillation-based approach to boost the accuracy of multilingual machine translation. Specifically, individual models are first trained and regarded as teachers, and then the multilingual model is trained to fit the training data and match the outputs of individual models simultaneously through knowledge distillation. Experiments on IWSLT, WMT and Ted talk translation datasets demonstrate the effectiveness of our method. Particularly, we show that one model is enough to handle multiple languages (up to 44 languages in our experiment), with comparable or even better accuracy than individual models.

Figures (1)

• Figure 1: The loss (Figure a) and BLEU score (Figure b: Ar-En, Figure c: Cs-En, Figure d: De-En) changes on the test set of the IWSLT dataset, with varying perturbation parameter $\sigma$.