On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis

TL;DR

This work analyzes the approximation and optimization behavior of continuous-time linear RNNs when learning temporal input-output mappings, focusing on linear functionals $H_t$ and memory kernels $ρ$. By leveraging a universal approximation framework and a kernel representation $H_t(\boldsymbol{x}) = \int_{0}^{\infty} x_{t-s}^T ρ(s) ds$, the authors show that linear RNNs can approximate broad classes of causal, regular, time-homogeneous functionals, with approximation quality tied to the memory decay of $ρ$. They derive rate results linking smoothness and memory to the required network width $m$, and reveal a memory-induced curse: slower decay of memory (smaller $ω$ in $ρ(t) ~ t^{-(1+ω)}$) necessitates exponentially more resources for approximation. In optimization, they reduce the problem to gradient-flow dynamics that exhibit plateaus when long-term memory is present, proving an exponential slowdown in training as memory grows, and corroborating with numerical experiments. Together, these results provide a rigorous lens on how temporal structure and memory shape both the expressivity and trainability of RNNs, offering guidance for mitigating long-term memory challenges in temporal learning tasks.

Abstract

We study the approximation properties and optimization dynamics of recurrent neural networks (RNNs) when applied to learn input-output relationships in temporal data. We consider the simple but representative setting of using continuous-time linear RNNs to learn from data generated by linear relationships. Mathematically, the latter can be understood as a sequence of linear functionals. We prove a universal approximation theorem of such linear functionals, and characterize the approximation rate and its relation with memory. Moreover, we perform a fine-grained dynamical analysis of training linear RNNs, which further reveal the intricate interactions between memory and learning. A unifying theme uncovered is the non-trivial effect of memory, a notion that can be made precise in our framework, on approximation and optimization: when there is long term memory in the target, it takes a large number of neurons to approximate it. Moreover, the training process will suffer from slow downs. In particular, both of these effects become exponentially more pronounced with memory - a phenomenon we call the "curse of memory". These analyses represent a basic step towards a concrete mathematical understanding of new phenomenon that may arise in learning temporal relationships using recurrent architectures.

Figures (1)

• Figure 1: Comparison of training dynamics on different types of functionals. (a) and (b): using the linear RNN model with the GD optimizer; (c): using the nonlinear RNN model (with $\tanh$ activation) with the Adam optimizer. The shaded region depicts the mean $\pm$ the standard deviation in 10 independent runs using randomized initialization. Observe that learning complex functionals (Airy, Lorenz) suffers from slow-downs in the form of long plateaus.

Theorems & Definitions (5)

• Definition 2.1
• Proposition 2.1
• Theorem 3.1: Universal approximation of linear functionals
• Theorem 3.2: Approximation rates of linear RNN
• Theorem 4.1