Kolmogorov-Arnold Networks in Low-Data Regimes: A Comparative Study with Multilayer Perceptrons

TL;DR

The paper addresses whether Kolmogorov-Arnold Networks (KANs) offer advantages in low-data regimes over traditional MLPs. It defines a fair comparison by enabling per-neuron trainable activations in MLPs and establishing a mathematical link between MLPs and KANs, then evaluates both architectures on synthetic data and two real-world datasets (cancer detection and 3D printer type prediction) to study depth and spline-order effects. The results show that MLPs with individualized activations achieve higher accuracy with only modest parameter increases, while KANs demand many more parameters and can underperform in data-scarce settings, especially as depth or spline order increases. These findings highlight activation-function design and parameter efficiency as critical factors for robust learning in low-data regimes and point to future directions involving alternative nonlinearities and per-neuron activation strategies.

Abstract

Multilayer Perceptrons (MLPs) have long been a cornerstone in deep learning, known for their capacity to model complex relationships. Recently, Kolmogorov-Arnold Networks (KANs) have emerged as a compelling alternative, utilizing highly flexible learnable activation functions directly on network edges, a departure from the neuron-centric approach of MLPs. However, KANs significantly increase the number of learnable parameters, raising concerns about their effectiveness in data-scarce environments. This paper presents a comprehensive comparative study of MLPs and KANs from both algorithmic and experimental perspectives, with a focus on low-data regimes. We introduce an effective technique for designing MLPs with unique, parameterized activation functions for each neuron, enabling a more balanced comparison with KANs. Using empirical evaluations on simulated data and two real-world data sets from medicine and engineering, we explore the trade-offs between model complexity and accuracy, with particular attention to the role of network depth. Our findings show that MLPs with individualized activation functions achieve significantly higher predictive accuracy with only a modest increase in parameters, especially when the sample size is limited to around one hundred. For example, in a three-class classification problem within additive manufacturing, MLPs achieve a median accuracy of 0.91, significantly outperforming KANs, which only reach a median accuracy of 0.53 with default hyperparameters. These results offer valuable insights into the impact of activation function selection in neural networks.

Figures (9)

• Figure 1: Illustrating the structural differences between Multilayer Perceptrons (MLPs) and Kolmogorov-Arnold Networks (KANs), highlighting that nonlinearities are introduced via the edges or connections between neurons in KANs.
• Figure 2: Visualization of the two simulated data sets with varying sample sizes (1,000 vs. 100 data points).
• Figure 3: Classification accuracy comparison: MLPs vs. KANs on simulated data sets A and B.
• Figure 4: Parameter count comparison: a substantial disparity in the total number of learnable parameters is observed between KANs and MLPs on simulated data.
• Figure 5: Evaluating the effect of depth (number of hidden layers) on classification accuracy: a comparative study of MLPs and KANs using the cancer detection data set with 569 samples.