Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks

Bo Chang, Minmin Chen, Eldad Haber, Ed H. Chi

TL;DR

This work reframes RNN trainability through a dynamical-systems lens and introduces AntisymmetricRNN, a recurrent architecture derived from discretizing an ODE with an antisymmetric hidden interaction to yield near-zero real parts of the Jacobian eigenvalues, thereby stabilizing gradients. By adding a diffusion term and a gating mechanism, the model preserves long-term memory while remaining computationally efficient compared to gating-heavy alternatives. Empirical results show strong performance on long-range memory tasks (e.g., permuted MNIST) with far fewer parameters than LSTMs, and competitive results on short-range CIFAR-10 tasks, with ablations highlighting the importance of the antisymmetric structure and diffusion. The approach opens a path to principled, well-conditioned recurrent architectures grounded in stability theory and numerical analysis.

Abstract

Recurrent neural networks have gained widespread use in modeling sequential data. Learning long-term dependencies using these models remains difficult though, due to exploding or vanishing gradients. In this paper, we draw connections between recurrent networks and ordinary differential equations. A special form of recurrent networks called the AntisymmetricRNN is proposed under this theoretical framework, which is able to capture long-term dependencies thanks to the stability property of its underlying differential equation. Existing approaches to improving RNN trainability often incur significant computation overhead. In comparison, AntisymmetricRNN achieves the same goal by design. We showcase the advantage of this new architecture through extensive simulations and experiments. AntisymmetricRNN exhibits much more predictable dynamics. It outperforms regular LSTM models on tasks requiring long-term memory and matches the performance on tasks where short-term dependencies dominate despite being much simpler.

AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks

TL;DR

This work reframes RNN trainability through a dynamical-systems lens and introduces AntisymmetricRNN, a recurrent architecture derived from discretizing an ODE with an antisymmetric hidden interaction to yield near-zero real parts of the Jacobian eigenvalues, thereby stabilizing gradients. By adding a diffusion term and a gating mechanism, the model preserves long-term memory while remaining computationally efficient compared to gating-heavy alternatives. Empirical results show strong performance on long-range memory tasks (e.g., permuted MNIST) with far fewer parameters than LSTMs, and competitive results on short-range CIFAR-10 tasks, with ablations highlighting the importance of the antisymmetric structure and diffusion. The approach opens a path to principled, well-conditioned recurrent architectures grounded in stability theory and numerical analysis.

Abstract

Recurrent neural networks have gained widespread use in modeling sequential data. Learning long-term dependencies using these models remains difficult though, due to exploding or vanishing gradients. In this paper, we draw connections between recurrent networks and ordinary differential equations. A special form of recurrent networks called the AntisymmetricRNN is proposed under this theoretical framework, which is able to capture long-term dependencies thanks to the stability property of its underlying differential equation. Existing approaches to improving RNN trainability often incur significant computation overhead. In comparison, AntisymmetricRNN achieves the same goal by design. We showcase the advantage of this new architecture through extensive simulations and experiments. AntisymmetricRNN exhibits much more predictable dynamics. It outperforms regular LSTM models on tasks requiring long-term memory and matches the performance on tasks where short-term dependencies dominate despite being much simpler.

Paper Structure

This paper contains 18 sections, 3 theorems, 22 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. AntisymmetricRNNs
  4. Ordinary differential equations
  5. Stability of ordinary differential equations: AntisymmetricRNNs
  6. Stability of the forward Euler method: Diffusion
  7. Gating mechanism
  8. Simulation
  9. Experiments
  10. Pixel-by-pixel MNIST
  11. Pixel-by-pixel CIFAR-10
  12. Noise padded CIFAR-10
  13. Ablation study
  14. Conclusion
  15. An overview of stability theory
  16. ...and 3 more sections

Key Result

Proposition 1

The solution of an ODE is stable if where $Re(\cdot)$ denotes the real part of a complex number.

Figures (5)

  • Figure 1: Visualization of the dynamics of RNNs and RNNs with feedback using different weight matrices.
  • Figure 2: Mean and standard deviation of eigenvalues of the end-to-end Jacobian matrix in AntisymmetricRNNs with different diffusion constants and LSTMs, trained on the noise padded CIFAR-10.
  • Figure 3: Visualization of the dynamics of vanilla RNNs with standard Gaussian random weights, using seeds from 1 to 16.
  • Figure 4: Visualization of the dynamics of RNN with feedback with standard Gaussian random weights, using seeds from 1 to 16, diffusion strength $\gamma=0.1$.
  • Figure 5: Visualization of the dynamics of RNN with feedback with independent standard Gaussian input.

Theorems & Definitions (5)

  • Definition 1
  • Proposition 1
  • Proposition 2
  • Proposition 3
  • proof