How noise affects memory in linear recurrent networks

TL;DR

Two major properties are revealed: first, the memory reduced by noise is uniquely determined by the noise's power spectral density (PSD), and second, the memory will not decrease regardless of noise intensity if the PSD is in a certain class of distribution (including power law).

Abstract

The effects of noise on memory in a linear recurrent network are theoretically investigated. Memory is characterized by its ability to store previous inputs in its instantaneous state of network, which receives a correlated or uncorrelated noise. Two major properties are revealed: First, the memory reduced by noise is uniquely determined by the noise's power spectral density (PSD). Second, the memory will not decrease regardless of noise intensity if the PSD is in a certain class of distribution (including power law). The results are verified using the human brain signals, showing good agreement.

Figures (3)

• Figure 1: The mechanism of memory reduction caused by autocorrelation of noise. The MF and MC of random input $u$ (purple) and two elements of autocorrelated noise $n$ (green) and $a$ (red) are illustrated. To demonstrate the mechanism, the superposition of sinusoidal wave and uniform random noise is used as a noise, whose ratios are $1$ to $0$, $1$ to $1$, and $0$ to $1$ in (a), (b), and (c), respectively. The input and all superposed noises have the same intensities. (a)--(c) show the MFs, whose vertical and horizontal axes are MF and delay, respectively. (d) sums up the upper MFs for each basis $u$, $n$, and $a$, whose horizontal and vertical axes are the label of noise and MC, respectively. All analyses were performed with a 20-node system and were repeated for 40 trials to average the MFs and MCs. (e) illustrates the MFs of the infinite-dimensional system with general noise, in which the MF of $u_{t-\tau}$ and $n_{t-\tau}$ are much longer than that of $a_{t-\tau}$.
• Figure 2: Dependency of normalized MC on $1/f$-scaling of noise and NSR. The color indicates $M_{\mathrm{sum},u} /N$, where $N=10^4$. The horizontal axis is the $1/f$-scaling $\beta$ of the noise and the vertical axis is NSR $r$.
• Figure 3: The effect of noise correlation in time series from the real world. The left (right) panels show MF (MC), whose horizontal axis is the delay $\tau$ of input (system size $N$). The solid (dotted) lines are calculated with the original (shuffled) EEG series obtained from three electrodes: Fz, Pz, and Cz. The left panel shows $M[u_{t-\tau}]$ in a 128-node RNN. The right panel shows $M_{\mathrm{sum},u}$ normalized by $M_{\rm sum}$, where the upper bound is the rank of system (gray dotted). All the plotted lines have been averaged over 40 trials. Across all noises, the intensities are the same value ($r=100$).