Multivariate time series prediction using clustered echo state network

TL;DR

The paper addresses the challenge of forecasting high-dimensional multivariate time series by proposing a clustered echo state network (CESN) that assigns each input variable to a dedicated reservoir cluster with dense intra-cluster and sparse inter-cluster connections. Each cluster has its own readout, trained via ridge regression, enabling independent yet integrated processing of multiple signals. Through systematic comparisons of ring, Erdős-Rényi, and scale-free topologies on real-world data (stock market NSE, NASA solar wind) and a chaotic Rössler system, CESNs consistently outperform conventional ESNs in accuracy and robustness, with ER and SF topologies delivering the strongest gains. The work demonstrates that modular reservoir design and carefully tuned inter-cluster connectivity can enhance predictive performance while maintaining computational efficiency, offering practical guidance for high-dimensional time series forecasting.

Abstract

Many natural and physical processes can be understood by analyzing multiple system variables evolving, forming a multivariate time series. Predicting such time series is challenging due to the inherent noise and interdependencies among variables. Echo state networks (ESNs), a class of Reservoir Computing (RC) models, offer an efficient alternative to conventional recurrent neural networks by training only the output weights while keeping the reservoir dynamics fixed, reducing computational complexity. We propose a clustered ESNs (CESNs) that enhances the ability to model and predict multivariate time series by organizing the reservoir nodes into clusters, each corresponding to a distinct input variable. Input signals are directly mapped to their associated clusters, and intra-cluster connections remain dense while inter-cluster connections are sparse, mimicking the modular architecture of biological neural networks. This architecture improves information processing by limiting cross-variable interference and enhances computational efficiency through independent cluster-wise training via ridge regression. We further explore different reservoir topologies, including ring, Erdős-Rényi (ER), and scale-free (SF) networks, to evaluate their impact predictive performance. Our algorithm works well across diverse real-world datasets such as the stock market, solar wind, and chaotic Rössler system, demonstrating that CESNs consistently outperform conventional ESNs in terms of predictive accuracy and robustness to noise, particularly when using ER and SF topologies. These findings highlight the adaptability of CESNs for complex, multivariate time series forecasting.

Figures (18)

• Figure 1: Basic architecture of the Echo state network
• Figure 2: Schematic representation of the CESN architecture, where inputs from a multivariate time series are directed to specific reservoir clusters. Each cluster processes different aspects of the input dynamics, enhancing feature extraction and representation.
• Figure 3: Illustration of different network topologies used in reservoir computing: (a) Conventional reservoir with randomly connected neurons, (b) clustered ring topology, (c) clustered ER network topology, (d) clustered BA topology. Each topology influences the reservoir's dynamical properties, affecting memory capacity and feature extraction in ESNs.
• Figure 4: Structure matrix plots illustrating connectivity patterns in different ESN architectures: (a) Unclustered ESN, where white dots represent neuron-to-neuron connections, and black regions indicate the absence of connections; (b) Clustered ring topology, (c) Clustered ER network, and (d) Clustered SF network. In (b–d), white represents inter-cluster connections, black denotes no connections, and red highlights intra-cluster connections. These structural variations influence the reservoir's computational efficiency and ability to capture complex temporal dependencies in multivariate time series data.
• Figure 5: Sensitivity analysis of the network performance with respect to the inter-cluster connection probability $P_{out} \in [0, 0.1]$ and intra-cluster connectivity $P_{in}$ (specific to each network topology). Results are shown for three network configurations: (a) Ring topology ($P_{in}$ = 1; hence, no error bars are present), (b) ER topology ($P_{in} \in [0.01, 1.0]$), and (c) SF topology (mean degree $g \in [2, 10]$). Each data point represents the mean normalized root mean square error ($\overline{\text{NRMSE}}$) computed over 50 independent realizations, with error bars indicating the corresponding standard deviation.