Efficient Neural Networks with Discrete Cosine Transform Activations

TL;DR

This work introduces Expressive Neural Networks (ENN), which replace fixed activations with learnable, DCT-based nonlinearities to achieve highly expressive yet compact models. By parameterizing each neuron's activation with trainable coefficients, ENN attains strong performance on binary classification and implicit neural representation tasks, while enabling straightforward pruning thanks to the orthogonal DCT basis. The paper provides a detailed complexity analysis and demonstrates a practical pruning strategy that safely removes up to 40% of activation coefficients with negligible degradation, underscoring the approach's efficiency and scalability. Overall, the ENN framework exemplifies how signal-processing concepts can be integrated into neural design to balance accuracy, compactness, and interpretability, with clear pathways for deployment in resource-constrained settings.

Abstract

In this paper, we extend our previous work on the Expressive Neural Network (ENN), a multilayer perceptron with adaptive activation functions parametrized using the Discrete Cosine Transform (DCT). Building upon previous work that demonstrated the strong expressiveness of ENNs with compact architectures, we now emphasize their efficiency, interpretability and pruning capabilities. The DCT-based parameterization provides a structured and decorrelated representation that reveals the functional role of each neuron and allows direct identification of redundant components. Leveraging this property, we propose an efficient pruning strategy that removes unnecessary DCT coefficients with negligible or no loss in performance. Experimental results across classification and implicit neural representation tasks confirm that ENNs achieve state-of-the-art accuracy while maintaining a low number of parameters. Furthermore, up to 40% of the activation coefficients can be safely pruned, thanks to the orthogonality and bounded nature of the DCT basis. Overall, these findings demonstrate that the ENN framework offers a principled integration of signal processing concepts into neural network design, achieving a balanced trade-off between expressiveness, compactness, and interpretability.

Figures (10)

• Figure 1: for the 6 hidden and output neurons in a binary classification task. Vertical lines indicate the input range where each neuron operates. The last figure shows the learned decision boundary.
• Figure 2: Bumps of the hidden neurons from Figure \ref{['fig:aaf_ring']}.
• Figure 3: Decision map of the for different values of $M_1$.
• Figure 4: Image predictions and ground truth on task 1.
• Figure 5: Image predictions and ground truth on task 2.