Growing Cosine Unit: A Novel Oscillatory Activation Function That Can Speedup Training and Reduce Parameters in Convolutional Neural Networks

TL;DR

This work investigates oscillatory activation functions as a route to faster training and smaller CNNs. It introduces the Growing Cosine Unit (GCU) with $C(z)=z\cos z$, proving that its infinite zeros yield multiple parallel decision boundaries, and demonstrates that a single GCU neuron can solve the XOR problem—a capability unavailable to monotone activations. The authors present two theorems on limitations of non-oscillatory activations, compare computational cost against Swish and Mish, and show that GCU offers faster convergence and improved accuracy on CIFAR-10, CIFAR-100, and Imagenette, including a notable 7% gain on Imagenette with a VGG-16 backbone. Visualization and gradient-flow analyses further reveal that GCU maintains healthier gradient dynamics and produces more confident feature detectors than ReLU-based counterparts. Overall, the paper argues that oscillatory activations can enhance gradient propagation and enable more compact networks, motivating broader exploration of such functions in deep learning.

Abstract

Convolutional neural networks have been successful in solving many socially important and economically significant problems. This ability to learn complex high-dimensional functions hierarchically can be attributed to the use of nonlinear activation functions. A key discovery that made training deep networks feasible was the adoption of the Rectified Linear Unit (ReLU) activation function to alleviate the vanishing gradient problem caused by using saturating activation functions. Since then, many improved variants of the ReLU activation have been proposed. However, a majority of activation functions used today are non-oscillatory and monotonically increasing due to their biological plausibility. This paper demonstrates that oscillatory activation functions can improve gradient flow and reduce network size. Two theorems on limits of non-oscillatory activation functions are presented. A new oscillatory activation function called Growing Cosine Unit(GCU) defined as $C(z) = z\cos z$ that outperforms Sigmoids, Swish, Mish and ReLU on a variety of architectures and benchmarks is presented. The GCU activation has multiple zeros enabling single GCU neurons to have multiple hyperplanes in the decision boundary. This allows single GCU neurons to learn the XOR function without feature engineering. Experimental results indicate that replacing the activation function in the convolution layers with the GCU activation function significantly improves performance on CIFAR-10, CIFAR-100 and Imagenette.

Figures (14)

• Figure 1: The XOR problem is the task of learning the XOR function with the smallest network. The red dots must be classified as positive (represented by +1) and the black dots must be classified as negative (represented by -1). A single line (hyperplane) cannot separate the two classes.
• Figure 2: Two solutions to the XOR problem learnt by a single neuron using the GCU activation function. Points in yellow were assigned a class label of +1 and points in blue were assigned a class label of -1 by a single GCU neuron.
• Figure 3: A single neuron solution to the XOR problem. A single neuron with the GCU activation function is capable of learning the XOR dataset shown in (\ref{['Eq: XOR dataset']}) exactly. The signum function at the output is used to map output values to $\pm 1$.
• Figure 4: Plot of different activation functions
• Figure 6: Average time over 1000 independent runs. Where each run consisted of applying the activation to a vector of length $10^{6}$ with elements uniformly distributed in the interval [-5 , 5].