ParalESN: Enabling parallel information processing in Reservoir Computing

TL;DR

ParalESN addresses RC scalability by introducing a parallelizable, diagonal complex recurrence coupled with a mixing layer, enabling associative-scan-based parallel computation and training of only the readout. It preserves the Echo State Property and universality for fading memory, and showing equivalence in expressivity to arbitrary linear reservoirs via diagonalization. Empirically, ParalESN achieves comparable accuracy to traditional RC on time-series benchmarks with substantial speedups, and outperforms RC baselines on 1-D pixel classification while demanding far less compute and energy than fully trainable models. The work offers a scalable, principled pathway for integrating reservoir computing into modern deep-learning pipelines.

Abstract

Reservoir Computing (RC) has established itself as an efficient paradigm for temporal processing. However, its scalability remains severely constrained by (i) the necessity of processing temporal data sequentially and (ii) the prohibitive memory footprint of high-dimensional reservoirs. In this work, we revisit RC through the lens of structured operators and state space modeling to address these limitations, introducing Parallel Echo State Network (ParalESN). ParalESN enables the construction of high-dimensional and efficient reservoirs based on diagonal linear recurrence in the complex space, enabling parallel processing of temporal data. We provide a theoretical analysis demonstrating that ParalESN preserves the Echo State Property and the universality guarantees of traditional Echo State Networks while admitting an equivalent representation of arbitrary linear reservoirs in the complex diagonal form. Empirically, ParalESN matches the predictive accuracy of traditional RC on time series benchmarks, while delivering substantial computational savings. On 1-D pixel-level classification tasks, ParalESN achieves competitive accuracy with fully trainable neural networks while reducing computational costs and energy consumption by orders of magnitude. Overall, ParalESN offers a promising, scalable, and principled pathway for integrating RC within the deep learning landscape.

Figures (6)

• Figure 1: Architectural organization of the proposed ParalESN. The model may have multiple blocks consisting of two components: (i) a linear reservoir and (ii) a non-linear mixing layer. The first block processes the external input as in a traditional shallow architecture. Subsequent blocks process the output of the previous block's mixing layer, $\mathbf{z}^{(\ell - 1)}_{\text{t}}$. The structure of the reservoir (in blue) includes a diagonal, complex-valued transition matrix $\mathbf{\Lambda}^{(\ell)}_{\text{h}}$ and a dense complex-valued input weight matrix $\mathbf{W}^{(\ell)}_{\text{in}}$. The main branch and the temporal residual connections are scaled by a positive coefficient $\tau$ and $1 - \tau$, respectively. The mixing layer (in purple) is used to introduce non-linearity in the model's dynamics and to mix the components of the reservoir states at each time step. The readout (in orange) is the only trainable component in the system. See Section \ref{['sec:paralesn']} for details.
• Figure 2: (left) Time required to perform the recurrence in ParalESN and traditional ESNs for increasing sequence lengths, assuming $128$ recurrent neurons and $5$ layers for deep configurations. ParalESN scales logarithmically with sequence length, whereas traditional ESNs scale linearly. (right) Scaling ParalESN and traditional ESNs to high-dimensional reservoirs on the sMNIST task. Traditional ESNs run out-of-memory (OOM) at approximately $100$K reservoir neurons, while ParalESN fit into memory due to the reduced memory footprint of their diagonal transition matrices.
• Figure 3: (left) Analysis of the trade-off between performance (error for regression tasks and accuracy for classification tasks) and efficiency (training time) for ParalESN and traditional ESNs across all considered benchmarks. For each model, we compute the percentage improvement over the ESN baseline for each task. The normalized scores are then obtained via min-max normalization of these average improvements, mapping them to a $[0, 1]$ scale, where $0$ corresponds to the worst-performing model and $1$ to the best-performing one. Overall, ParalESN and ParalESN (deep) outperform their counterparts while being more efficient. (right) Critical difference plot computed via a Wilcoxon test demvsar2006statistical, showing the average rank (lower is better). Models are ranked based on their overall performance across all benchmarks. Cliques (solid lines) connect models for which there is no statistically significant difference in performance. On average, ParalESN (deep) is the top-performing model.
• Figure 4: Training time comparison between ParalESNs and traditional ESNs for each memory-based and forecasting benchmark. ParalESN and ParalESN (deep) train order of magnitude faster.
• Figure 5: Trade-off between performance (test accuracy) and efficiency (training time) for ParalESN, traditional RC, and fully-trainable sequence models, for the sMNIST and psMNIST benchmarks. For each model and benchmark, we compute the percentage improvement over the ESN baseline. The normalized scores are then obtained via min-max normalization of these average improvements, mapping them to a $[0, 1]$ scale with $0$ representing to the worst-performing model and $1$ the best-performing one. ParalESN is competitive with fully-trainable models while delivering substantial efficiency improvements.

Theorems & Definitions (4)

• Theorem 4.1: Sufficient and necessary conditions for the ESP
• Proposition 4.2
• Corollary 4.3
• Definition 4.1