Comparison of Reservoir Computing topologies using the Recurrent Kernel approach

TL;DR

The Recurrent Kernel limit of all these RC topologies is defined and a convergence study for a wide range of activation functions and hyperparameters is conducted and it is demonstrated that there is an optimal sparsity level which grows with the reservoir size.

Abstract

Reservoir Computing (RC) has become popular in recent years thanks to its fast and efficient computational capabilities. Standard RC has been shown to be equivalent in the asymptotic limit to Recurrent Kernels, which helps in analyzing its expressive power. However, many well-established RC paradigms, such as Leaky RC, Sparse RC, and Deep RC, are yet to be systematically analyzed in such a way. We define the Recurrent Kernel limit of all these RC topologies and conduct a convergence study for a wide range of activation functions and hyperparameters. Our findings provide new insights into various aspects of Reservoir Computing. First, we demonstrate that there is an optimal sparsity level which grows with the reservoir size. Furthermore, our analysis suggests that Deep RC should use reservoir layers of decreasing sizes. Finally, we perform a benchmark demonstrating the efficiency of Structured Reservoir Computing compared to vanilla and Sparse Reservoir Computing.

Figures (6)

• Figure 1: Recurrent Kernels associated with various Reservoir Computing topologies. RC, sparse RC, and structured RC converge to the same RK limit when the reservoir size $N \rightarrow \infty$. Leaky RC and Deep RC converge to their corresponding limits.
• Figure 2: Convergence study of various Reservoir Computing topologies (columns) towards their corresponding Recurrent Kernel limits, for different activation functions $f$ (rows). For each case, the Frobenius norm between the RC and RK Gram matrices $L$ (Eq. \ref{['eq: metric convergence']}, smaller is better) is displayed for weight scaling factors $\sigma_r$, $\sigma_i$ between 0.04 and 2. Blue dot in the first row: typical operating point $\sigma_r = \sigma_i = 1$.
• Figure 3: (Top) Error metric (Eq. \ref{['eq: metric convergence']}) normalized between 0 and 1 as a function of sparsity for different reservoir sizes. (Bottom) Sparsity threshold above which the error metric is within $5\%$ of the non-sparse limit. This gives an admissible sparsity level which decreases with the reservoir size.
• Figure 4: Error metric (Eq. \ref{['eq: metric convergence']}) for Deep Reservoir Computing for varying reservoir sizes. We vary the first layer size $N_1$ for a fixed computational budget $N_1^2+N_2^2 = 2 \times N_\text{med}^2$ for different values of the median reservoir size $N_\text{med}$.
• Figure 5: Time benchmark of the matrix-vector multiplication on CPU and GPU for Reservoir Computing, Sparse Reservoir Computing, and Dense Reservoir Computing.