ANAct: Adaptive Normalization for Activation Functions

TL;DR

The paper addresses gradient-variance degradation caused by activation functions by deriving a general variance expression and proposing ANAct, a dynamic, mini-batch–driven normalization that stabilizes forward and backward gradient flow. By enforcing that both $\rho_i$ and $\rho_i'$ stay near unity through a learned normalization factor $\lambda_i$, the method improves convergence and generalization across CNNs, ResNets, and NLP models, with NSwish and NReLU delivering notable gains over their unnormalized counterparts. The approach demonstrates practical benefits, including reduced parameter counts and improved compatibility with Batch Normalization and residual architectures, across tasks such as MNIST, CIFAR-100, Tiny ImageNet, IMDb, and IWSLT translations. Overall, ANAct offers a theoretically motivated, broadly applicable technique to enhance training stability and accuracy when using activation functions in deep networks.

Abstract

In this paper, we investigate the negative effect of activation functions on forward and backward propagation and how to counteract this effect. First, We examine how activation functions affect the forward and backward propagation of neural networks and derive a general form for gradient variance that extends the previous work in this area. We try to use mini-batch statistics to dynamically update the normalization factor to ensure the normalization property throughout the training process, rather than only accounting for the state of the neural network after weight initialization. Second, we propose ANAct, a method that normalizes activation functions to maintain consistent gradient variance across layers and demonstrate its effectiveness through experiments. We observe that the convergence rate is roughly related to the normalization property. We compare ANAct with several common activation functions on CNNs and residual networks and show that ANAct consistently improves their performance. For instance, normalized Swish achieves 1.4\% higher top-1 accuracy than vanilla Swish on ResNet50 with the Tiny ImageNet dataset and more than 1.2\% higher with CIFAR-100.

Figures (6)

• Figure 1: The upper panel and the lower panel are two different activation function architectures we compare: (a) Vanilla Activation Function (b) Normalized Activation Function
• Figure 2: As shown above, the $\mathcal{R}$ score of common activation functions are constantly below 0. The $\mathcal{R}$ score of Tanh and Swish is desirably near 0 when the variance of input is small. However, they suffer from pseudo-linearity DBLP:conf/iclr/PhilippSC18 at the same time. The $\mathcal{R}$ score of ReLU and LeakyReLU are fixed at -0.383. Data are calculated with Mathematica.
• Figure 3: The left figure shows the increasing validation accuracy.The right figure illustrates the scores that we defined in Eq \ref{['eq:convergence-score']}. Each curve is the median of 25 runs.
• Figure 4: We recorded $\sum_i^N \left|\mathbb{E}{\bm{W}}_i\right|$ during training. The curves are the median of 25 runs.
• Figure 5: Validation accuracy of VGG19 on CIFAR-100. All curves are the median of 3 runs.