Residual Reservoir Memory Networks

TL;DR

This work introduces Residual Reservoir Memory Networks (ResRMNs), a dual-reservoir recurrent architecture that couples a linear memory reservoir with a non-linear ResESN module using temporal residual connections to improve long-range information propagation. Stability is analyzed via a Jacobian-based linearization, yielding a necessary condition that both the linear memory and the non-linear residual components remain stable at the origin, while the linear module is kept at unit spectral radius to operate at the edge of stability. Empirical evaluation on time-series classification and pixel-level 1-D tasks shows that ResRMNs typically outperform baselines like LeakyESN and other RC variants, with the identity-orthogonal configuration often providing the best results. The approach demonstrates that a carefully designed dual-reservoir RC can achieve strong performance with untrained reservoirs, offering a flexible, hardware-friendly path for sequential data processing; future work includes exploring alternative initializations and deeper spectral analyses.

Abstract

We introduce a novel class of untrained Recurrent Neural Networks (RNNs) within the Reservoir Computing (RC) paradigm, called Residual Reservoir Memory Networks (ResRMNs). ResRMN combines a linear memory reservoir with a non-linear reservoir, where the latter is based on residual orthogonal connections along the temporal dimension for enhanced long-term propagation of the input. The resulting reservoir state dynamics are studied through the lens of linear stability analysis, and we investigate diverse configurations for the temporal residual connections. The proposed approach is empirically assessed on time-series and pixel-level 1-D classification tasks. Our experimental results highlight the advantages of the proposed approach over other conventional RC models.

Figures (5)

• Figure 1: The architecture of a Residual Reservoir Memory Network (ResRMN), assuming the hyperbolic tangent $tanh$ as the activation function and an orthogonal matrix $\mathbf{O}$ in the residual branch of the Residual Echo State Network (ResESN). The architecture consists of two untrained components: (1) a linear memory reservoir driven by the external input $\mathbf{x}$, and (2) a non-linear residual reservoir driven by both the external input $\mathbf{x}$ and the output of the memory reservoir $\mathbf{m}$. The final output is fed to a readout layer, which is the only trainable component.
• Figure 2: Eigenvalues of the different orthogonal matrices considered in the non-linear module, assuming a hidden size of $N_{h} = 100$. In orange the unitary circle.
• Figure 3: Eigenvalues of the Jacobian for different ResRMN configurations. The dynamics are driven by a random input vector and a random state, both uniformly distributed in $(-1, 1)$. We assume hidden sizes $N_{m}, N_{h} = 100$, a spectral radius $\rho = 1$, all input weight matrices with scaling of $1$ (i.e., $\omega_{x}, \omega_{x_{m}}, \omega_{m} = 1$), zero bias $\omega_{b} = 0$, and scaling coefficients $\alpha, \beta = 1$. In red the $N_{m}$ eigenvalues of the linear memory module, in blue the $N_{h}$ eigenvalues of the ResESN module. In orange the unitary circle.
• Figure 4: (a) Performance change in time-series classification tasks relative to the number of memory reservoir neurons $N_{m}$, with the base case being $N_{m} = T$. Results are broken down for non-residual and residual reservoir memory networks and averaged across all datasets. (b) Performance change in time-series classification tasks relative to LeakyESN. Results are broken down by model class and averaged across all datasets.
• Figure 5: Test results on psMNIST for a varying number of trainable parameters. For RMN and ResRMNs, the number of memory reservoir neurons $N_{m}$ is fixed to $784$. Reported results represent the average, and the corresponding standard deviation, across $5$ random initializations.

Theorems & Definitions (2)

• Theorem 1
• proof