Three Decades of Activations: A Comprehensive Survey of 400 Activation Functions for Neural Networks

TL;DR

The paper delivers a comprehensive compilation of real-valued activation functions, synthesizing three decades of research into a single, navigable resource. It distinguishes fixed versus adaptive AFs, surveys classical families (binary, sigmoid, ReLU and their variants), and catalogs an expansive set of sigmoid-weighted, gated, square-based, polynomial, and chaos-inspired activations, including hundreds of named variants. By providing precise definitions and links to original sources, it aims to curb duplication and accelerate exploration for researchers selecting AFs for neural networks. The work emphasizes breadth and organization over benchmarking, positioning the catalog as a foundational reference to guide principled AF selection and future adaptations. The resource supports practitioners and theorists alike by clarifying construction principles, adaptability, and trade-offs in AF design, enabling faster progress in neural network training and deployment.

Abstract

Neural networks have proven to be a highly effective tool for solving complex problems in many areas of life. Recently, their importance and practical usability have further been reinforced with the advent of deep learning. One of the important conditions for the success of neural networks is the choice of an appropriate activation function introducing non-linearity into the model. Many types of these functions have been proposed in the literature in the past, but there is no single comprehensive source containing their exhaustive overview. The absence of this overview, even in our experience, leads to redundancy and the unintentional rediscovery of already existing activation functions. To bridge this gap, our paper presents an extensive survey involving 400 activation functions, which is several times larger in scale than previous surveys. Our comprehensive compilation also references these surveys; however, its main goal is to provide the most comprehensive overview and systematization of previously published activation functions with links to their original sources. The secondary aim is to update the current understanding of this family of functions.