Exploring the Relationship: Transformative Adaptive Activation Functions in Comparison to Other Activation Functions

TL;DR

This work addresses the rapid proliferation of activation functions in neural networks by providing a comprehensive, consolidated survey of roughly 400 real-valued activation functions, categorized into fixed and adaptive types. It contextualizes existing literature and benchmark efforts, discusses methods for discovering or selecting AFs (including grid and evolutionary search) and highlights trade-offs in computational efficiency. While not performing exhaustive benchmarks, it positions the Transformative Adaptive Activation Functions (TAAFs) within this landscape as a flexible and versatile component, and argues that the consolidated resource will reduce redundant proposals and accelerate progress in AF research. Overall, the paper offers a foundational reference that supports informed AF choice and future innovations in neural network nonlinearity.

Abstract

Neural networks are the state-of-the-art approach for many tasks and the activation function is one of the main building blocks that allow such performance. Recently, a novel transformative adaptive activation function (TAAF) allowing for any vertical and horizontal translation and scaling was proposed. This work sets the TAAF into the context of other activation functions. It shows that the TAAFs generalize over 50 existing activation functions and utilize similar concepts as over 70 other activation functions, underscoring the versatility of TAAFs. This comprehensive exploration positions TAAFs as a promising and adaptable addition to neural networks.