Don't Fear Peculiar Activation Functions: EUAF and Beyond

TL;DR

The paper tackles the gap between the theoretical appeal of super-expressive activation functions and their practical utility. It generalizes EUAF to a broader analytic+periodic activation class and introduces PEUAF, a trainable-frequency variant, then proves a density result showing that such activations can approximate any continuous function in fixed-size networks when combined with Kolmogorov’s superposition theorem. Through extensive experiments on industrial (1D) and image (2D) datasets, PEUAF demonstrates improved convergence and fault localization, and when used in mixed-activation configurations, often yields superior performance on vision tasks. The work offers a path to harnessing super-expressiveness in real-world models, suggesting broader applicability and guiding principles for designing trainable, expressive activations with theoretical guarantees.

Abstract

In this paper, we propose a new super-expressive activation function called the Parametric Elementary Universal Activation Function (PEUAF). We demonstrate the effectiveness of PEUAF through systematic and comprehensive experiments on various industrial and image datasets, including CIFAR10, Tiny-ImageNet, and ImageNet. Moreover, we significantly generalize the family of super-expressive activation functions, whose existence has been demonstrated in several recent works by showing that any continuous function can be approximated to any desired accuracy by a fixed-size network with a specific super-expressive activation function. Specifically, our work addresses two major bottlenecks in impeding the development of super-expressive activation functions: the limited identification of super-expressive functions, which raises doubts about their broad applicability, and their often peculiar forms, which lead to skepticism regarding their scalability and practicality in real-world applications.

Figures (12)

• Figure 1: An illustration of $\mathrm{EUAF}$.
• Figure 2: Illustrations of $\varrho_1$, $\varrho_2$, and $\varrho_3$.
• Figure 3: An illustration of ${\mathcal{I}}_k$ and ${\widetilde{\mathcal{I}}}_k$ for $k\in\{1,2, \mathinner{\mkern-0.1mu\cdotp\mkern-0.3mu\cdotp\mkern-0.3mu\cdotp\mkern-0.1mu},K\}$ with $K=5$.
• Figure 4: An illustration of $\psi$ on $[0,10]$.
• Figure 5: An illustration of $\psi$ on $[0,2K]$ for $K=5$.

Theorems & Definitions (5)

• Theorem 1
• Lemma 1: Proposition 10 of JMLR:v25:23-0912
• Lemma 2: Lemma 23 of 2021Deep
• Lemma 3
• proof