Non-Autoregressive Machine Translation with Auxiliary Regularization

TL;DR

This work tackles the efficiency gap in non-autoregressive translation by introducing two training-time regularizations that enhance decoder hidden representations, addressing repeated and incomplete translations without increasing inference cost. The similarity regularization makes adjacent hidden states reflect the semantic distance between consecutive target tokens, while reconstruction regularization enforces source-side information to be recoverable from decoder states via a backward autoregressive model. Together, these methods yield state-of-the-art NAT performance on major benchmarks (e.g., 24.61 BLEU En-De, 28.90 BLEU De-En on WMT14) and substantial speedups, highlighting a practical path to fast, accurate translation without discrete latent variables. The results demonstrate the potential of regularization-based enhancements to NAT and suggest applicability to other sequence-generation tasks.

Abstract

As a new neural machine translation approach, Non-Autoregressive machine Translation (NAT) has attracted attention recently due to its high efficiency in inference. However, the high efficiency has come at the cost of not capturing the sequential dependency on the target side of translation, which causes NAT to suffer from two kinds of translation errors: 1) repeated translations (due to indistinguishable adjacent decoder hidden states), and 2) incomplete translations (due to incomplete transfer of source side information via the decoder hidden states). In this paper, we propose to address these two problems by improving the quality of decoder hidden representations via two auxiliary regularization terms in the training process of an NAT model. First, to make the hidden states more distinguishable, we regularize the similarity between consecutive hidden states based on the corresponding target tokens. Second, to force the hidden states to contain all the information in the source sentence, we leverage the dual nature of translation tasks (e.g., English to German and German to English) and minimize a backward reconstruction error to ensure that the hidden states of the NAT decoder are able to recover the source side sentence. Extensive experiments conducted on several benchmark datasets show that both regularization strategies are effective and can alleviate the issues of repeated translations and incomplete translations in NAT models. The accuracy of NAT models is therefore improved significantly over the state-of-the-art NAT models with even better efficiency for inference.

Figures (1)

• Figure 1: The overall architecture of NAT model with auxiliary regularization terms. AT stands for autoregressive translation. Each sublayer of the encoder/decoder contains a residual connection and layer normalization following transformer.