An Empirical Study of Generation Order for Machine Translation

TL;DR

The paper investigates whether generation order matters in machine translation by leveraging an insertion-based Insertion Transformer and a soft order-reward framework that enables training under arbitrary oracle orders. It systematically evaluatesuninformed, location-based, frequency-based, content-based, and model-based orders on WMT'14 En-De and WMT'18 En-Zh, comparing against a standard Transformer baseline. For English-German, most orders yield BLEU scores close to the Transformer, indicating order is largely non-essential; English-Chinese results vary more, showing sensitivity to generation order and opportunity for tailored ordering. The work demonstrates flexible, partially autoregressive generation and provides a principled training approach to explore diverse generation schemas, opening avenues for non-monotonic generation in MT and related sequence tasks.

Abstract

In this work, we present an empirical study of generation order for machine translation. Building on recent advances in insertion-based modeling, we first introduce a soft order-reward framework that enables us to train models to follow arbitrary oracle generation policies. We then make use of this framework to explore a large variety of generation orders, including uninformed orders, location-based orders, frequency-based orders, content-based orders, and model-based orders. Curiously, we find that for the WMT'14 English $\to$ German translation task, order does not have a substantial impact on output quality, with unintuitive orderings such as alphabetical and shortest-first matching the performance of a standard Transformer. This demonstrates that traditional left-to-right generation is not strictly necessary to achieve high performance. On the other hand, results on the WMT'18 English $\to$ Chinese task tend to vary more widely, suggesting that translation for less well-aligned language pairs may be more sensitive to generation order.

Figures (6)

• Figure 1: A schematic of the Insertion Transformer model for a Chinese-English translation pair. The model is encouraged to predict the correct set of remaining words within each slot. Using our order-reward framework (Section \ref{['sec:reward-framework']}), we can derive the necessary weight distribution to apply to the set of correct actions in order to train the model to follow any oracle generation policy of interest.
• Figure 2: Example decoding paths for serial and parallel generation using the Insertion Transformer.
• Figure 3: Example decodes for models trained to generate tokens in alphabetical (Unicode) order. Blue tokens correspond those being inserted at the current time step, and gray tokens correspond to those not yet generated. Note that the desired ordering applies on a per-slot basis rather than a global basis.
• Figure 4: An example of longest-first generation.
• Figure 5: An example of common-first generation.