Hysteresis Activation Function for Efficient Inference

TL;DR

This work addresses the dying ReLU problem by introducing HeLU, a hysteresis-based activation with a backward-threshold offset α that preserves forward ReLU behavior while maintaining nonzero gradients during training. The method incurs no additional inference cost and shows competitive or superior performance to ReLU across CIFAR-10/100, Imagenette, and GLUE benchmarks, often approaching GELU. It also demonstrates improved inference efficiency in transformer models, particularly under quantization. The results suggest that a simple training-time modification to activation functions can yield meaningful throughput gains for hardware-friendly neural networks, with future work on selecting α and extending to more domains.

Abstract

The widely used ReLU is favored for its hardware efficiency, {as the implementation at inference is a one bit sign case,} yet suffers from issues such as the ``dying ReLU'' problem, where during training, neurons fail to activate and constantly remain at zero, as highlighted by Lu et al. Traditional approaches to mitigate this issue often introduce more complex and less hardware-friendly activation functions. In this work, we propose a Hysteresis Rectified Linear Unit (HeLU), an efficient activation function designed to address the ``dying ReLU'' problem with minimal complexity. Unlike traditional activation functions with fixed thresholds for training and inference, HeLU employs a variable threshold that refines the backpropagation. This refined mechanism allows simpler activation functions to achieve competitive performance comparable to their more complex counterparts without introducing unnecessary complexity or requiring inductive biases. Empirical evaluations demonstrate that HeLU enhances model generalization across diverse datasets, offering a promising solution for efficient and effective inference suitable for a wide range of neural network architectures.

Figures (5)

• Figure 1: The threshold transition of bit voltage between 0 and 1 (forward) and 1 and 0 (backward) inspired the concept of using different thresholds for activation functions in the forward and backward passes of the network.
• Figure 2: Weight distribution of all pre-activations of Wide ResNet 40-4. In the image classification task with over parametric network, we see that majority of the pre-activation features are negative. In red we mark the pre-activations that we allow regret with HeLU.
• Figure 3: ReLU, GELU and HeLU functions and their derivatives
• Figure 4: Comparing HeLU and GELU in BERT. Top figures are Inference time by dataset, using half precision and QINT8 versions. Bottom figures are Throughput using half precision and QINT8.
• Figure 5: Analysis of various transformer architectures inference time per sample, using their original activation and HeLU.

Theorems & Definitions (2)

• Definition 1: HeLU
• Definition 2: HeLU Modified Derivative