A Generalized Framework of Sequence Generation with Application to Undirected Sequence Models

TL;DR

This work introduces a generalized probabilistic framework for sequence generation that unifies decoding from directed and undirected neural sequence models by separately modeling length, coordinate selection, and symbol replacement. By instantiating this framework, it recovers monotonic autoregressive, non-autoregressive iterative refinement, and semi-autoregressive decoding as special cases, and enables transfer of decoding methods to masked language models like BERT. The authors design both handcrafted and learned decoding strategies, including adaptive coordinate selection and length-conditioned beam search, and demonstrate competitive performance on WMT'14 English-German translation with a masked translation model. Importantly, they show constant-time decoding can approach linear-time performance under appropriate strategies, highlighting practical benefits for fast generation without severely sacrificing quality. The results suggest that undirected models, when paired with suitable decoding schemes, can match strong autoregressive baselines and enable flexible generation orders and efficient inference.

Abstract

Undirected neural sequence models such as BERT (Devlin et al., 2019) have received renewed interest due to their success on discriminative natural language understanding tasks such as question-answering and natural language inference. The problem of generating sequences directly from these models has received relatively little attention, in part because generating from undirected models departs significantly from conventional monotonic generation in directed sequence models. We investigate this problem by proposing a generalized model of sequence generation that unifies decoding in directed and undirected models. The proposed framework models the process of generation rather than the resulting sequence, and under this framework, we derive various neural sequence models as special cases, such as autoregressive, semi-autoregressive, and refinement-based non-autoregressive models. This unification enables us to adapt decoding algorithms originally developed for directed sequence models to undirected sequence models. We demonstrate this by evaluating various handcrafted and learned decoding strategies on a BERT-like machine translation model (Lample & Conneau, 2019). The proposed approach achieves constant-time translation results on par with linear-time translation results from the same undirected sequence model, while both are competitive with the state-of-the-art on WMT'14 English-German translation.

Figures (10)

• Figure 1: Generation orders given by easy-first, least2most, and learned coordinate selection. We use greedy search with $L$ iterations on the development set. We group the orders into five clusters using and visualize cluster centers with normalized positions (x-axis) over normalized generation steps (y-axis). The thickness of a line is proportional to the number of examples in the corresponding cluster.
• Figure 2: Average difference in energy $\uparrow$ between sequences generated by selecting positions uniformly at random versus by different algorithms, over the course of decoding.
• Figure 3: Evolution of the energy of the sequence $\downarrow$ over the course of decoding by different position selection algorithms.
• Figure 4: Example sentences generated following an right-to-left-to-right-to-left generation order, using the easy-first decoding algorithm on De$\rightarrow$En.
• Figure 5: Example sentences generated following an outside-in generation order, using the easy-first decoding algorithm on De$\rightarrow$En.