Paradoxical noise preference in RNNs

TL;DR

This work investigates why continuous-time recurrent neural networks (CTRNNs) often exhibit peak performance at a nonzero training noise level, rather than at zero. By analyzing function computation, maze navigation, and a single-neuron regulator, the authors show that noise shifts the mean of the neural state’s stationary distribution, biasing outputs when noise is removed. The bias is amplified when neural states approach activation nonlinearities, causing networks to operate near boundaries between linear regions and thereby making the training noise an integral part of the computation rather than a mere regularizer. Distinguishing this effect from stochastic resonance, the study suggests that noise-induced fixed-point shifts can shape neural population dynamics and inform the design of robust RNNs.

Abstract

In recurrent neural networks (RNNs) used to model biological neural networks, noise is typically introduced during training to emulate biological variability and regularize learning. The expectation is that removing the noise at test time should preserve or improve performance. Contrary to this intuition, we find that continuous-time recurrent neural networks (CTRNNs) often perform best at a nonzero noise level, specifically, the same level used during training. This noise preference typically arises when noise is injected inside the neural activation function; networks trained with noise injected outside the activation function perform best with zero noise. Through analyses of simple function approximation, maze navigation, and single neuron regulator tasks, we show that the phenomenon stems from noise-induced shifts of fixed points (stationary distributions) in the underlying stochastic dynamics of the RNNs. These fixed point shifts are noise-level dependent and bias the network outputs when the noise is removed, degrading performance. Analytical and numerical results show that the bias arises when neural states operate near activation function nonlinearities, where noise is asymmetrically attenuated, and that performance optimization incentivizes operation near these nonlinearities. Thus, networks can overfit to the stochastic training environment itself rather than just to the input-output data. The phenomenon is distinct from stochastic resonance, wherein nonzero noise enhances signal processing. Our findings reveal that training noise can become an integral part of the computation learned by recurrent networks, with implications for understanding neural population dynamics and for the design of robust artificial RNNs.

Figures (6)

• Figure 1: A continuous-time recurrent neural network with input and output at each time. Variants of this network are used in computational neuroscience yang2019taskdriscoll2024flexible, and the effect of synaptic noise on such networks is studied here.
• Figure 2: Simple function computation: Noise-in networks prefer the training noise level, but noise-out networks prefer zero noise. a) Root mean squared error, mean error, and error standard deviations for noise-in function computation networks. The best performance occurs when testing at or near the training noise level, and this preference is driven by a non-monotonic dependence between the noise level and the mean error (i.e., the systematic error). This indicates that passing the noise through the activation function somehow induces a noise-level-dependent systematic bias in the network outputs. b) Schematic illustrating that the network output trajectories with training noise has the lower total error compared to when noise is zero or higher. The mean trajectory is depicted by a thick line and individual realizations by thin lines. c) Root mean squared error, mean error, and error standard deviations for noise-out function computation networks. Unlike the noise-in networks, the noise-out networks perform best when tested with zero noise, and the systematic error appears unaffected by changes in the noise.
• Figure 3: Maze navigation task and results. a) Schematic of the maze environment. The blue x's show the locations of maze vertices where the network has to stop and wait as it navigates from the start vertex to the destination vertex. The red arrows between vertices denote the multiple short jaunts that make up one of the routes through the maze. b) Root mean squared error, mean error, and error standard deviation for a noise-in maze navigation network when tested at a variety of different noise levels. As with the function computation networks, the network performs best at the training noise level, and the performance deficits at other noise levels are driven by error bias. For instance, setting noise to zero during test time results in the particle systematically falling short of its destination.
• Figure 4: Noise can shift the fixed points of both noise-in and noise-out networks if they have fixed points near a boundary between linear regions of their dynamics. a) Functional relationship between the mean of a normal distribution before and after transformation by the ReLU activation function for multiple different standard deviations. When the absolute value of the mean is on the order of the standard deviation, interaction with the nonlinearity causes a variance-dependent positive shift in the mean of the transformed distribution. As a result, the locations of fixed points in a noise-in network will depend on the noise level if those fixed points are situated near boundaries between linear regions in the dynamics; a no-noise single-neuron version of such a noise-in network is shown as an inset. b) Functional relationship between the mean of the stationary distribution of a piecewise Ornstein-Uhlenbeck process (Equations \ref{['OUprocess']} and \ref{['OUrnn']}) and the standard deviation of the noise. Inset shows a no-noise single-neuron version of such a noise-out network. As the standard deviation grows, the stationary distribution shifts in the positive direction because deviations in that direction take longer to decay. This single-neuron example illustrates why the locations of fixed points in a noise-out network depend on the noise level if those fixed points are situated near boundaries between linear regions.
• Figure 5: Noise-induced fixed point shifts predict input-specific error in noise-in function computation networks tested with zero noise. Noise induced fixed point shifts projected onto the output matrix closely track the zero noise error profile across the input domain, confirming that noise-induced fixed point shifts are responsible for the noise preference.