Impact of white Gaussian internal noise on analog echo-state neural networks

TL;DR

This work analyzes how internal white Gaussian noise affects hardware-like analog ESNs across reservoir topologies and four noise configurations (additive/multiplicative × uncorrelated/correlated). It derives analytic expressions for output-variance under noise in both the output layer and the reservoir, showing that variance scales with statistics of the output weight matrix as $ ext{Var}[y^{out}] ightarrow$ terms involving $N^2 ext{mu}^2(oldsymbol{W}^{out})$ and $N ext{eta}(oldsymbol{W}^{out})$, as well as reservoir-state variability. A key finding is that non-self-closed ESNs exhibit linear growth of error with noise intensity, while self-closed ESNs can experience rapid and severe signal loss due to noise feedback, with additive correlated noise being the most destructive. The study also shows that activation functions strongly influence noise propagation through the reservoir via $oldsymbol{W}^{out}$ statistics, with linear activations being particularly fragile, and suggests that biasing may mitigate some noise effects in hardware ESNs.

Abstract

In recent years, more and more works have appeared devoted to the analog (hardware) implementation of artificial neural networks, in which neurons and the connection between them are based not on computer calculations, but on physical principles. Such networks offer improved energy efficiency and, in some cases, scalability, but may be susceptible to internal noise. This paper studies the influence of noise on the functioning of recurrent networks using the example of trained echo state networks (ESNs). The most common reservoir connection matrices were chosen as various topologies of ESNs: random uniform and band matrices with different connectivity. White Gaussian noise was chosen as the influence, and according to the way of its introducing it was additive or multiplicative, as well as correlated or uncorrelated. In the paper, we show that the propagation of noise in reservoir is mainly controlled by the statistical properties of the output connection matrix, namely the mean and the mean square. Depending on these values, more correlated or uncorrelated noise accumulates in the network. We also show that there are conditions under which even noise with an intensity of $10^{-20}$ is already enough to completely lose the useful signal. In the article we show which types of noise are most critical for networks with different activation functions (hyperbolic tangent, sigmoid and linear) and if the network is self-closed.

Figures (8)

• Figure 1: Schematic representation of considered ESN (a) and how it can be used to predict chaotic time series. Panels (b,d,f) show the original time series (blue), prediction of ESN (black) and prediction of self-closed ESN (orange). These panels were prepared for three trained ESNs with random uniform $\mathbf{W}^\mathrm{res}$ (b,c) and band matrices with $5\%$ connectivity (d,e) and $10\%$ connectivity (f,g).
• Figure 2: Variance of the output ESN signal $y^\mathrm{out}_t$ in the case of noise in the output layer for three trained ESNs with random uniform $\mathbf{W}^\mathrm{res}$ (a), band matrix with 5% connectivity (b) and band matrix with 10% connectivity (c). Blue points correspond to additive noise with $D^C_A=0.001$, while red points were prepared for multiplicative noise with $D^C_M=0.001$. The analytics estimation of the variance level (black lines) were obtained using (\ref{['eq:var_out']}).
• Figure 3: Averaged MSE (a) and variance (b) of the output ESN signal $y^\mathrm{out}_t$ in the case of noise in the output layer of trained ESNs with uniform $\mathbf{W}^\mathrm{res}$. Blue color correspond to additive noise, while all dependences for multiplicative noise are shown in red. Coloured backgrounds in (b) show the ranges of variance.
• Figure 4: Variance of the output ESN signal $y^\mathrm{out}_t$ in the case of noise in reservoir for three trained ESNs with different $\mathbf{W}^\mathrm{res}$. Four noise sources were considered: correlated additive (blue) and multiplicative (red); uncorrelated additive (purple) and multiplicative (orange). Black dashed lines show the analytical prediction of variance levels based on (\ref{['eq:var_corr_noise_res']}). All noise intensities are the same $D^U_A=D^U_M=D^C_A=D^C_M=10^{-3}$.
• Figure 5: Averaged MSE (a) and variance (b) of the output ESN signal $y^\mathrm{out}_t$ in the case of noise in the reservoir of trained ESNs with uniform $\mathbf{W}^\mathrm{res}$. Four noise sources were considered: correlated additive (blue) and multiplicative (red); uncorrelated additive (purple) and multiplicative (orange). Coloured backgrounds in (b) show the ranges of variance.