Deep Reinforcement Learning For Sequence to Sequence Models

TL;DR

The paper surveys how reinforcement learning can address fundamental training deficiencies in seq2seq models, notably exposure bias and misalignment between training objectives and discrete evaluation metrics. By framing seq2seq generation as a policy-learning problem, it surveys policy-gradient, actor-critic, and advanced Q-learning methods, and how they have been integrated with seq2seq architectures (including pointer-generator and attention-based models). It highlights practical framework options, discusses current challenges such as sample efficiency and reward design, and presents an open-source RLSeq2Seq library with experiments on abstractive summarization. Overall, the work guides researchers in selecting suitable RL strategies to enhance long-horizon sequence generation and provides a practical toolkit for experimenting with RL-informed seq2seq training.

Abstract

In recent times, sequence-to-sequence (seq2seq) models have gained a lot of popularity and provide state-of-the-art performance in a wide variety of tasks such as machine translation, headline generation, text summarization, speech to text conversion, and image caption generation. The underlying framework for all these models is usually a deep neural network comprising an encoder and a decoder. Although simple encoder-decoder models produce competitive results, many researchers have proposed additional improvements over these sequence-to-sequence models, e.g., using an attention-based model over the input, pointer-generation models, and self-attention models. However, such seq2seq models suffer from two common problems: 1) exposure bias and 2) inconsistency between train/test measurement. Recently, a completely novel point of view has emerged in addressing these two problems in seq2seq models, leveraging methods from reinforcement learning (RL). In this survey, we consider seq2seq problems from the RL point of view and provide a formulation combining the power of RL methods in decision-making with sequence-to-sequence models that enable remembering long-term memories. We present some of the most recent frameworks that combine concepts from RL and deep neural networks and explain how these two areas could benefit from each other in solving complex seq2seq tasks. Our work aims to provide insights into some of the problems that inherently arise with current approaches and how we can address them with better RL models. We also provide the source code for implementing most of the RL models discussed in this paper to support the complex task of abstractive text summarization.

Figures (4)

• Figure 1: A simple seq2seq model. The blue boxes correspond to the encoder part which has $T_e$ units. The green boxes correspond to the decoder part which has $T$ units.
• Figure 2: A simple attention-based pointer-generation seq2seq model with Self-Critic reward. At each decoding step, the context vector for that decoder is calculated and combined with the decoder output to get the action distribution. In pointer-generation model, the attention distribution is further combined with the action distribution through switches called pointers to get the final distribution over the actions. From each output distribution, a specific action $\hat{y}_2$ is sampled and the greedy action $\hat{y}^{g}_2$ is extracted. The difference of the rewards from sampling and greedy sequence is used to update the loss function.
• Figure 3: A simple Actor-Critic model with an attention-based pointer-generation seq2seq model as the Actor. The Critic model is shown on the right side of the picture with a purple box. The purple box $A_{\Psi}$, which represents the Critic model, takes as input the decoder output at time $t=2$, i.e., $s_2$, and estimate the advantage values through either (value function estimation, DQN, DDQN, or dueling net) for each action.
• Figure 4: A list of advantages and drawbacks of different RL models. The advantages are listed such that each method covers all the strengths of its previous methods and drawbacks are listed such that each method have all the weaknesses of the previous ones. For instance, Actor-Critic w. Dueling Net have all the pros of the previous models listed above it and Actor-Critic w. Value Function Estimation suffers from all the cons of the methods listed below it. The features that are also model-dependent are shown with '$*$' and those features do not exist in any other model. Each '$\star$' shows how hard it is to implement these models in a real-world application.