Noisy Activation Functions

TL;DR

The paper tackles training challenges from saturating activation functions by introducing noise into activations, enabling gradient flow even in saturated regimes. By learning a noise schedule and optionally annealing it, the method behaves like a continuation approach, improving optimization and performance across tasks such as language modeling, machine translation, and image captioning. The approach provides both input- and output-based noise variants, yields faster convergence, and often attains state-of-the-art results, validating noisy activations as a versatile training aid for difficult objectives.

Abstract

Common nonlinear activation functions used in neural networks can cause training difficulties due to the saturation behavior of the activation function, which may hide dependencies that are not visible to vanilla-SGD (using first order gradients only). Gating mechanisms that use softly saturating activation functions to emulate the discrete switching of digital logic circuits are good examples of this. We propose to exploit the injection of appropriate noise so that the gradients may flow easily, even if the noiseless application of the activation function would yield zero gradient. Large noise will dominate the noise-free gradient and allow stochastic gradient descent toexplore more. By adding noise only to the problematic parts of the activation function, we allow the optimization procedure to explore the boundary between the degenerate (saturating) and the well-behaved parts of the activation function. We also establish connections to simulated annealing, when the amount of noise is annealed down, making it easier to optimize hard objective functions. We find experimentally that replacing such saturating activation functions by noisy variants helps training in many contexts, yielding state-of-the-art or competitive results on different datasets and task, especially when training seems to be the most difficult, e.g., when curriculum learning is necessary to obtain good results.

Figures (11)

• Figure 1: The plot for derivatives of different activation functions.
• Figure 2: An example of a one-dimensional, non-convex objective function where a simple gradient descent will behave poorly. With large noise $|\xi| \rightarrow \infty$, SGD can escape from saddle points and bad local-minima as a result of exploration. As we anneal the noise level $|\xi| \rightarrow 0$, SGD will eventually converge to one of the local-minima $x^{\ast}$.
• Figure 3: A simple depiction of adding Gaussian noise on the linearized activation function, which brings the average back to the hard-saturating nonlinearity $\text{h}(x)$, in bold. Its linearization is $\text{u}(x)$ and the noisy activation is $\phi$. The difference $\text{h}(x)-\text{u}(x)$ is $\Delta$ which is a vector indicates the discrepancy between the linearized function and the actual function that the noise is being added to $\text{h}(x)$. Note that, $\Delta$ will be zero, at the non-saturating parts of the function where $\text{u}(x)$ and $\text{h}(u)$ matches perfectly.
• Figure 4: Stochastic behavior of the proposed noisy activation function with different $\alpha$ values and with noise sampled from the Normal distribution, approximating the $\operatorname{hard-tanh}$ nonlinearity (in bold green).
• Figure 5: Learning curves of a single layer MLP trained with RMSProp with different noise types and activation functions

Theorems & Definitions (4)

• Definition 2.1
• Definition 2.2
• Definition 2.3
• Remark