Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Recurrent Dropout without Memory Loss

Stanislau Semeniuta, Aliaksei Severyn, Erhardt Barth

TL;DR

Recurrent Dropout without Memory Loss addresses regularization for gated RNNs by dropping the recurrent update vectors rather than hidden states, mitigating memory-loss issues. The authors propose per-step mask sampling and demonstrate that dropping g_t preserves memory and improves generalization when combined with forward dropout. They empirically validate on synthetic tasks and NLP benchmarks including language modeling, NER, and sentiment classification, showing consistent gains over prior recurrent dropout schemes. The work provides practical guidance on where to apply dropout in LSTMs/GRUs and sampling strategy, advancing robust RNN regularization in NLP.

Abstract

This paper presents a novel approach to recurrent neural network (RNN) regularization. Differently from the widely adopted dropout method, which is applied to \textit{forward} connections of feed-forward architectures or RNNs, we propose to drop neurons directly in \textit{recurrent} connections in a way that does not cause loss of long-term memory. Our approach is as easy to implement and apply as the regular feed-forward dropout and we demonstrate its effectiveness for Long Short-Term Memory network, the most popular type of RNN cells. Our experiments on NLP benchmarks show consistent improvements even when combined with conventional feed-forward dropout.

Recurrent Dropout without Memory Loss

TL;DR

Recurrent Dropout without Memory Loss addresses regularization for gated RNNs by dropping the recurrent update vectors rather than hidden states, mitigating memory-loss issues. The authors propose per-step mask sampling and demonstrate that dropping g_t preserves memory and improves generalization when combined with forward dropout. They empirically validate on synthetic tasks and NLP benchmarks including language modeling, NER, and sentiment classification, showing consistent gains over prior recurrent dropout schemes. The work provides practical guidance on where to apply dropout in LSTMs/GRUs and sampling strategy, advancing robust RNN regularization in NLP.

Abstract

This paper presents a novel approach to recurrent neural network (RNN) regularization. Differently from the widely adopted dropout method, which is applied to \textit{forward} connections of feed-forward architectures or RNNs, we propose to drop neurons directly in \textit{recurrent} connections in a way that does not cause loss of long-term memory. Our approach is as easy to implement and apply as the regular feed-forward dropout and we demonstrate its effectiveness for Long Short-Term Memory network, the most popular type of RNN cells. Our experiments on NLP benchmarks show consistent improvements even when combined with conventional feed-forward dropout.

Paper Structure

This paper contains 12 sections, 19 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Recurrent Dropout
  4. Dropout in vanilla RNNs
  5. Dropout in LSTM networks
  6. Experiments
  7. Synthetic Task
  8. Word Level Language Modeling
  9. Character Level Language Modeling
  10. Named Entity Recognition
  11. Twitter Sentiment Analysis
  12. Conclusions

Figures (2)

  • Figure 1: Illustration of the three types of dropout in recurrent connections of LSTM networks. Dashed arrows refer to dropped connections. Input connections are omitted for clarity.
  • Figure 2: Learning curves of LSTM networks when training without and with 0.25 per-step recurrent dropout. Solid and dashed lines show training and validation errors respectively. Best viewed in color.