Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Recent Advances in Recurrent Neural Networks

Hojjat Salehinejad, Sharan Sankar, Joseph Barfett, Errol Colak, Shahrokh Valaee

TL;DR

<3-5 sentence high-level summary> The paper surveys recurrent neural networks (RNNs), detailing their fundamentals, training challenges such as vanishing/exploding gradients, and a comprehensive taxonomy of architectures from simple RNNs to LSTMs, GRUs, and memory-augmented variants. It reviews optimization and regularization techniques, including BPTT, SGD/Adam, Hessian-free methods, and dropout-type schemes, highlighting how these address training stability and generalization. The survey covers a wide range of architectures (BRNN, MDLSTM, Grid LSTM, SCRN, unitary/orthogonal RNNs) and their applications across text, speech, image, and video domains, illustrating both successes and open challenges. It also discusses potential directions such as unitary/orthogonal RNNs, deeper integration with external memory, and domain-specific extensions to 3D data and multimedia signals.

Abstract

Recurrent neural networks (RNNs) are capable of learning features and long term dependencies from sequential and time-series data. The RNNs have a stack of non-linear units where at least one connection between units forms a directed cycle. A well-trained RNN can model any dynamical system; however, training RNNs is mostly plagued by issues in learning long-term dependencies. In this paper, we present a survey on RNNs and several new advances for newcomers and professionals in the field. The fundamentals and recent advances are explained and the research challenges are introduced.

Recent Advances in Recurrent Neural Networks

TL;DR

<3-5 sentence high-level summary> The paper surveys recurrent neural networks (RNNs), detailing their fundamentals, training challenges such as vanishing/exploding gradients, and a comprehensive taxonomy of architectures from simple RNNs to LSTMs, GRUs, and memory-augmented variants. It reviews optimization and regularization techniques, including BPTT, SGD/Adam, Hessian-free methods, and dropout-type schemes, highlighting how these address training stability and generalization. The survey covers a wide range of architectures (BRNN, MDLSTM, Grid LSTM, SCRN, unitary/orthogonal RNNs) and their applications across text, speech, image, and video domains, illustrating both successes and open challenges. It also discusses potential directions such as unitary/orthogonal RNNs, deeper integration with external memory, and domain-specific extensions to 3D data and multimedia signals.

Abstract

Recurrent neural networks (RNNs) are capable of learning features and long term dependencies from sequential and time-series data. The RNNs have a stack of non-linear units where at least one connection between units forms a directed cycle. A well-trained RNN can model any dynamical system; however, training RNNs is mostly plagued by issues in learning long-term dependencies. In this paper, we present a survey on RNNs and several new advances for newcomers and professionals in the field. The fundamentals and recent advances are explained and the research challenges are introduced.

Paper Structure

This paper contains 52 sections, 58 equations, 14 figures, 4 tables.

Table of Contents

  1. Introduction
  2. A Simple Recurrent Neural Network
  3. Model Architecture
  4. Activation Function
  5. Loss Function
  6. Training Recurrent Neural Network
  7. Initialization
  8. Gradient-based Learning Methods
  9. Back-propagation through time (BPTT)
  10. Vanishing Gradient Problem
  11. Exploding Gradient Problem
  12. Stochastic Gradient Descent
  13. Mini-Batch Gradient Descent
  14. Adam Stochastic Optimization
  15. Extended Kalman Filter-based Learning
  16. ...and 37 more sections

Figures (14)

  • Figure 1: A simple recurrent neural network (RNN) and its unfolded structure through time $t$. Each arrow shows a full connection of units between the layers. To keep the figure simple, biases are not shown.
  • Figure 2: Most common activation functions.
  • Figure 3: As the network is receiving new inputs over time, the sensitivity of units decay (lighter shades in layers) and the back-propagation through time (BPTT) overwrites the activation in hidden units. This results in forgetting the early visited inputs.
  • Figure 4: The classical momentum and the Nesterov accelerated gradient schemes.
  • Figure 5: Some deep recurrent neural network (RNN) architectures with multi-layer perceptron (MLP).
  • ...and 9 more figures