Ensemble Distillation for Neural Machine Translation

TL;DR

The paper investigates knowledge distillation for Neural Machine Translation (NMT), showing that a single student model can reproduce the quality of a six-model ensemble while offering faster decoding. It introduces two teacher paradigms (ensemble and oracle Bleu) and a data-filtering strategy that prunes noisy training data by leveraging teacher-driven translations. By translating the entire training set with the teacher and optionally adding these translations as training targets, the authors achieve substantial quality gains, even with the student sharing the same architecture as the teacher. They also demonstrate that an ensemble teacher can train a smaller student that matches or exceeds baseline performance, improving practicality for real-time translation tasks on German→English data. The results highlight practical routes to faster, smaller, and higher-quality NMT systems without code changes, with strong improvements on the WMT 2016 task.

Abstract

Knowledge distillation describes a method for training a student network to perform better by learning from a stronger teacher network. Translating a sentence with an Neural Machine Translation (NMT) engine is time expensive and having a smaller model speeds up this process. We demonstrate how to transfer the translation quality of an ensemble and an oracle BLEU teacher network into a single NMT system. Further, we present translation improvements from a teacher network that has the same architecture and dimensions of the student network. As the training of the student model is still expensive, we introduce a data filtering method based on the knowledge of the teacher model that not only speeds up the training, but also leads to better translation quality. Our techniques need no code change and can be easily reproduced with any NMT architecture to speed up the decoding process.