Simply Trainable Nearest Neighbour Machine Translation with GPU Inference

TL;DR

The paper tackles domain adaptation in neural machine translation by enhancing kNN-MT with a simply trainable single-layer interpolation that automatically predicts the interpolation weight $λ$ per domain. It constructs a compact, input-adaptive datastore via BM25 and trains a one-layer network to combine the MT and kNN distributions, achieving improvements in BLEU and COMET across multiple domains while requiring only about 40 minutes of single-GPU training. The method integrates with GPU inference through FasterTransformer and demonstrates a small speed penalty (~5–7%) on large MoE models, making live-domain adaptation practical. Overall, the approach provides an effective, automatic domain-adaptation mechanism for large-scale MT systems with minimal additional latency.

Abstract

Nearest neighbor machine translation is a successful approach for fast domain adaption, which interpolates the pre-trained transformers with domain-specific token-level k-nearest-neighbor (kNN) retrieval without retraining. Despite kNN MT's success, searching large reference corpus and fixed interpolation between the kNN and pre-trained model led to computational complexity and translation quality challenges. Among other papers, Dai et al. proposed methods to obtain a small number of reference samples dynamically for which they introduced a distance-aware interpolation method using an equation that includes free parameters. This paper proposes a simply trainable nearest neighbor machine translation and carry out inference experiments on GPU. Similar to Dai et al., we first adaptively construct a small datastore for each input sentence. Second, we train a single-layer network for the interpolation coefficient between the knnMT and pre-trained result to automatically interpolate in different domains. Experimental results on different domains show that our proposed method either improves or sometimes maintain the translation quality of methods in Dai et al. while being automatic. In addition, our GPU inference results demonstrate that knnMT can be integrated into GPUs with a drop of only 5% in terms of speed.