Expanded Gating Ranges Improve Activation Functions

TL;DR

The paper tackles the limitations of conventional self-gated activation functions that constrain gating to $[0,1]$ by introducing expanded gating ranges via a trainable scalar $ hoa$, applied to ArcTan Linear Unit (ATLU) and extended to xGELU, xSiLU, and GLU variants. Using OpenWebText2-scale transformer experiments, the authors show that xATLU, xGELU, and xSiLU outperform GELU and SiLU, with xATLU delivering the strongest gains due to its monotonically increasing first derivative and enhanced negative gradient flow. They also reveal that expanded gating ranges bridge the performance gap between first- and second-order GLUs, though second-order GLUs do not consistently benefit from expansion. The findings advance activation-function design by evidencing that gating-range expansion can yield meaningful improvements in training dynamics and model performance, with implications for broader neural architectures and future search for new gating functions.

Abstract

Activation functions are core components of all deep learning architectures. Currently, the most popular activation functions are smooth ReLU variants like GELU and SiLU. These are self-gated activation functions where the range of the gating function is between zero and one. In this paper, we explore the viability of using arctan as a gating mechanism. A self-gated activation function that uses arctan as its gating function has a monotonically increasing first derivative. To make this activation function competitive, it is necessary to introduce a trainable parameter for every MLP block to expand the range of the gating function beyond zero and one. We find that this technique also improves existing self-gated activation functions. We conduct an empirical evaluation of Expanded ArcTan Linear Unit (xATLU), Expanded GELU (xGELU), and Expanded SiLU (xSiLU) and show that they outperform existing activation functions within a transformer architecture. Additionally, expanded gating ranges show promising results in improving first-order Gated Linear Units (GLU).

Figures (4)

• Figure 1: Comparison of ATLU, GELU, and SiLU.(a) Graph of gating functions for ATLU, GELU, and SiLU. All are continuously differentiable, monotonically increasing, and have a gating range of $(0, 1)$. (b) Graph of ATLU, GELU, and SiLU. ATLU differs from GELU and SiLU in that it is not ReLU-like: it does not converge to 0 as $x$ approaches negative infinity and does not converge to $x$ as $x$ approaches positive infinity. (c) Graph of first derivatives for ATLU, GELU, and SiLU. The first derivative of ATLU is monotonically increasing and does not have values below 0 or above 1, unlike GELU and SiLU.
• Figure 2: Visualisation of xATLU. (a) Graph of xATLU for various fixed values of $\alpha$. Increasing $\alpha$ makes the activation function converge towards more positive values in both directions. A similar effect occurs when applied to GELU and SiLU. (b) Graph of the first derivative of xATLU for various fixed values of $\alpha$. Increasing $\alpha$ increases the range of the first derivative. A similar effect occurs when applied to GELU and SiLU.
• Figure 3: Effect of Expanded Gating Ranges. Experiments analysing the effect of using a fixed scalar value for $\alpha$. Note that using trainable scalar values performs better. The baseline activation functions ATLU, GELU and SiLU are at $\alpha=0$. Increasing $\alpha$ improves the performance of xATLU, xGELU and xSiLU, allowing them to surpass the performance of GELU and SiLU.
• Figure 4: Comparison of trainable $\alpha$ weights. We use a heatmap to visualize learned $\alpha$ weights for depth 48 transformer models trained on OpenWebText2 using xATLU, xGELU and xSiLU. Note that we place no restrictions on the values that $\alpha$ can take and the learned values are all positive, which means $\alpha$ is expanding the gating ranges.