Leveraging Continuously Differentiable Activation Functions for Learning in Quantized Noisy Environments

TL;DR

The paper addresses learning in real-world analog and noisy hardware, where quantization and noise disrupt gradient propagation. It argues that continuously differentiable activations like GELU and SiLU offer more stable gradient flow than ReLU, and formalizes this with EP, RP, and GSD metrics, plus an interpolation framework between ReLU and differentiable activations. Across CNNs (ConvNet, VGG-A, ResNet-18) and Vision Transformers, it demonstrates that GELU/SiLU improve convergence and accuracy under quantized noise, with larger gains as model depth increases. The findings provide practical hardware-guidance for activation selection in photonic and other analog accelerators, enabling more reliable, high-performance low-precision AI systems.

Abstract

Real-world analog systems intrinsically suffer from noise that can impede model convergence and accuracy on a variety of deep learning models. We demonstrate that differentiable activations like GELU and SiLU enable robust propagation of gradients which help to mitigate analog quantization error that is ubiquitous to all analog systems. We perform analysis and training of convolutional, linear, and transformer networks in the presence of quantized noise. Here, we are able to demonstrate that continuously differentiable activation functions are significantly more noise resilient over conventional rectified activations. As in the case of ReLU, the error in gradients are 100x higher than those in GELU near zero. Our findings provide guidance for selecting appropriate activations to realize performant and reliable hardware implementations across several machine learning domains such as computer vision, signal processing, and beyond. Code available at: \href{https://github.com/Vivswan/GeLUReLUInterpolation}{https://github.com/Vivswan/GeLUReLUInterpolation}.}

Figures (7)

• Figure 1: Overview of model architecture.a) For the Linear, Convolutional, VGG and ResNet models we assume the worst case scenario where both the sensor and the model are physical and exhibit analog noise. For instance, this is the case for a CCD exhibiting electronic noise in conjunction with an analog photonic networks for computation. This is done by adding quantized noise layers between each traditional layer of the model. Adapted from shah_analogvnn_2023. b) For a Vision Transformer model, the sensor could be implemented in analog hardware while the transformer network is implemented in digital hardware. c) ReLU, GELU and SiLU activation function and its derivatives
• Figure 2: Effects of Scaling Factor in GELU. GELU function and its derivative at different values of scaling factor with a) with full precision; b) with reduced precision. c) The effective bit-precision of GELU derivative near zero at different values of scaling factor when input precision is set to 6-bits. d) Top-1 test accuracy of ConvNet marginally declines with increasing GELU scaling factor on CIFAR-10.
• Figure 3: Interpolation Factors. ReLU-GELU interpolation function and its derivative at different values of interpolation factor: a) at full-precision b) with reduced precision; c) with reduced precision and noise. d) The Gradient Step Discontinuity in ReLU-GELU interpolation at zero is negatively correlated to interpolation factor. e) & f) Top-1 Test Accuracy of ConvNet Utilizing Linear Interpolation Activation Functions, (e) ReLU-GELU and (f) ReLU-SiLU, with Quantized Noise on CIFAR-10 Dataset.
• Figure 4: Gradient Error in ReLU and GELU when inputs and weight are 8-bit quantized with 0.5 error probability for noise in a linear layer with no bias followed by an activation layer.a) shows the gradient errors when interpolating between ReLU and GELU. Gradient Error b) for ReLU activation; c) for GELU activations;. NOTE: the gradient error colorbar axis is different for ReLU and GELU in (B) and (C) respectively.
• Figure 5: Models evaluated on the CIFAR-100 dataset. ReLU-GELU interpolation function and its derivative at different values of interpolation factor for ConvNet, VGG-A and ResNet-18 models on CIFAR-100