Attention-Enhanced Reservoir Computing

TL;DR

This work introduces an attention-enhanced reservoir computing (AERC) framework for photonic delay-based reservoirs to improve chaotic time-series prediction. By replacing the traditional readout with an attention mechanism whose weights are generated by a learned neural mapping from reservoir states, the approach enables dynamic prioritization of informative temporal features. Empirical results on unidirectionally-coupled Lorenz and alternating Lorenz–Rössler tasks show that AERC achieves lower NRMSE and longer valid prediction times, especially for smaller reservoirs, and can capture key spectral components more accurately than linear ridge regression. The findings suggest a practical path toward high-performance, hardware-friendly chaotic time-series forecasting with potential for full photonic implementations and real-time applications.

Abstract

Photonic reservoir computing has been successfully utilized in time-series prediction as the need for hardware implementations has increased. Prediction of chaotic time series remains a significant challenge, an area where the conventional reservoir computing framework encounters limitations of prediction accuracy. We introduce an attention mechanism to the reservoir computing model in the output stage. This attention layer is designed to prioritize distinct features and temporal sequences, thereby substantially enhancing the prediction accuracy. Our results show that a photonic reservoir computer enhanced with the attention mechanism exhibits improved prediction capabilities for smaller reservoirs. These advancements highlight the transformative possibilities of reservoir computing for practical applications where accurate prediction of chaotic time series is crucial.

Figures (14)

• Figure 1: (a) : Self-attention mechanism creating queries ( $q$$\textbf{q}$ ), keys ( $k$$\textbf{k}$ ), and values ( $v$$\textbf{v}$ ) from an input. (b) : Reservoir approach transforming an input into a high dimensional reservoir state $R(t)$$\textbf{R}(t)$ . Both projections are then used in a neural network an abstract function $\alpha$ to yield the attention weights $W_{att,n}$$\textbf{w}_{att}$ .
• Figure 2: Schematic representation of the attention-enhanced reservoir computing process augmented setup. The trainable parameters $\textbf{W}_{\text{net}}$ take the reservoir node states and yield the attention weights $\textbf{w}_{\text{att}}$, which are then weighted with the reservoir node states to generate an attention mechanismoutput .
• Figure 3: Schematic representation of the reservoir computing process augmented with an attention mechanism.
• Figure 4: Example time series of open-loop configuration for the UCTLS. (a) shows the x-variable of the time series, and (b) the attention weights in the same time interval for a 50-node reservoir. The color bar shows the magnitude of the attention weights. The dashed black lines indicate a shift from the attractor, pinpointing the dynamical adjustment of the weights to the task.
• Figure 5: Example time series of closed-loop configuration for the UCTLS. (a) shows the x-variable of the time series, and (b) the attention weights in the same time interval for a 50-node reservoir. The true trajectory and closed-loop prediction are depicted by the solid blue and dashed black lines, respectively. The color bar shows the magnitude of the attention weights. The vertical dashed black line indicates the time point for VPT computation.