Biologically Inspired Oscillating Activation Functions Can Bridge the Performance Gap between Biological and Artificial Neurons

TL;DR

This work addresses the limited expressive power of single artificial neurons by leveraging oscillating activation functions inspired by pyramidal neurons, which possess multiple zeros and can realize the XOR boundary with a single unit. It introduces four activations—Shifted Quadratic Unit (SQU), Non-Monotonic Cubic (NCU), Shifted Sinc Unit (SSU), and Decaying Sine Unit (DSU)—and demonstrates that such activations yield multi-hyperplane decision boundaries, faster gradient flow, and superior training efficiency. Through extensive experiments on CIFAR-10, CIFAR-100, and Imagenette, the oscillating activations consistently outperform 23 baselines, often placing in the top ranks and enabling accurate results with fewer layers. The findings suggest that biologically inspired oscillations can partially bridge the gap between biological and artificial neural networks, offering a practical path to more expressive and efficient models.

Abstract

The recent discovery of special human neocortical pyramidal neurons that can individually learn the XOR function highlights the significant performance gap between biological and artificial neurons. The output of these pyramidal neurons first increases to a maximum with input and then decreases. Artificial neurons with similar characteristics can be designed with oscillating activation functions. Oscillating activation functions have multiple zeros allowing single neurons to have multiple hyper-planes in their decision boundary. This enables even single neurons to learn the XOR function. This paper proposes four new oscillating activation functions inspired by human pyramidal neurons that can also individually learn the XOR function. Oscillating activation functions are non-saturating for all inputs unlike popular activation functions, leading to improved gradient flow and faster convergence. Using oscillating activation functions instead of popular monotonic or non-monotonic single-zero activation functions enables neural networks to train faster and solve classification problems with fewer layers. An extensive comparison of 23 activation functions on CIFAR 10, CIFAR 100, and Imagentte benchmarks is presented and the oscillating activation functions proposed in this paper are shown to outperform all known popular activation functions.

Figures (9)

• Figure 1: A single neuron solution to the XOR problem using a Shifted Quadratic Unit (SQU).
• Figure 2: A single neuron solution to the XOR problem using a Non-Monotonic Cubic Unit (NCU).
• Figure 3: A single neuron solution to the XOR problem using a Shifted Sinc Unit (SSU).
• Figure 4: Plot of (a) activation function and (b) derivatives.
• Figure 5: Training performance on CIFAR-10.