Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Information

TL;DR

This work tackles the persistent quality gap between non-autoregressive NMT and autoregressive models by introducing ReorderNAT, a framework that explicitly models target-side reordering through a dedicated reordering module that generates a pseudo-translation. The decoder then translates conditioned on this pseudo-translation, using either deterministic or non-deterministic guiding strategies to narrow the decoding search space and better capture target dependencies. Empirical results across multiple language pairs demonstrate consistent improvements over existing NAT approaches and substantial speedups, with particularly strong gains when word order differs, such as in Chinese–English. The study shows that reordering-informed NAT can approach autoregressive performance while preserving much of NAT’s speed advantages, offering a practical path toward efficient, high-quality MT systems.

Abstract

Non-autoregressive neural machine translation (NAT) generates each target word in parallel and has achieved promising inference acceleration. However, existing NAT models still have a big gap in translation quality compared to autoregressive neural machine translation models due to the enormous decoding space. To address this problem, we propose a novel NAT framework named ReorderNAT which explicitly models the reordering information in the decoding procedure. We further introduce deterministic and non-deterministic decoding strategies that utilize reordering information to narrow the decoding search space in our proposed ReorderNAT. Experimental results on various widely-used datasets show that our proposed model achieves better performance compared to existing NAT models, and even achieves comparable translation quality as autoregressive translation models with a significant speedup.

Figures (6)

• Figure 1: The architecture of our ReorderNAT model. Different from original NAT models, our model adds a reordering module between the encoder and decoder modules to explicitly model the reordering information. For original NAT models, the decoder inputs are the copied embeddings of source sentence (No.1 dashed arrow), and for our ReorderNAT model, the decoder inputs are the embeddings of pseudo-translation generated by reordering module (No. 2 dashed arrow). The encoder and decoder blocks are the same as existing NMT models (e.g., Transformer block).
• Figure 2: Translation quality on the IWSLT16 validation set over various input sentence lengths.
• Figure 3: The framework of our ReorderNAT model.
• Figure 4: Translation quality on the WMT14 and WMT 16 test sets over various input sentence lengths.
• Figure 5: Effect of the Transformer decoder block number in the decoder module on the IWSLT16 validation set.