DropKAN: Regularizing KANs by masking post-activations

TL;DR

KANs rely on trainable edge activations and node-wise post-sums, which makes traditional Dropout ineffective due to nonzero post-activation propagation and non-homogeneous activations. DropKAN embeds a binary mask directly into the KAN computation, with post-activation masking (pa) and post-spline masking (ps), and uses scaling to preserve expected activity. Theoretical analysis clarifies why standard Dropout fails for KANs, and empirical results on ten UCI datasets show DropKAN consistently improves generalization over No-Drop and Dropout, with the pa variant closely matching No-Drop in forward passes. The approach is simple to implement and generalizable to other KAN-based architectures, offering a practical regularization tool for KANs with spline or similar activations $\phi_{j,i}(x_i)= w_b b(x_i) + w_s\,spline(x_i)$.

Abstract

We propose DropKAN (Dropout Kolmogorov-Arnold Networks) a regularization method that prevents co-adaptation of activation function weights in Kolmogorov-Arnold Networks (KANs). DropKAN functions by embedding the drop mask directly within the KAN layer, randomly masking the outputs of some activations within the KANs' computation graph. We show that this simple procedure that require minimal coding effort has a regularizing effect and consistently lead to better generalization of KANs. We analyze the adaptation of the standard Dropout with KANs and demonstrate that Dropout applied to KANs' neurons can lead to unpredictable behavior in the feedforward pass. We carry an empirical study with real world Machine Learning datasets to validate our findings. Our results suggest that DropKAN is consistently a better alternative to using standard Dropout with KANs, and improves the generalization performance of KANs. Our implementation of DropKAN is available at: \url{https://github.com/Ghaith81/dropkan}.

Figures (2)

• Figure 1: A standard KAN network (on the left), a KAN network regularized using Dropout applied between the KAN layers by masking neurons/nodes, with dashed gray lines indicating zero inputs to the connected activations (middle), a KAN network regularized using DropKAN by masking post-activations, where dashed gray lines indicate zero inputs to the connected neurons (right).
• Figure 2: The expected value of the output neuron ($x_{3,1}$) of the [6, 2, 2 ,1] KAN with No-Drop using the car dataset compared to DropKAN$^{pa}_{w/ scale}$ and DropKAN$^{pa}_{w/o\ scale}$ (on the left), and to Dropout$_{w/ scale}$ and Dropout$_{w/o\ scale}$ (right) at different stages of the training.