Kolmogorov-Arnold Convolutions: Design Principles and Empirical Studies

TL;DR

The work investigates integrating Kolmogorov-Arnold Networks (KANs) into convolutional architectures for computer vision, introducing Bottleneck KAN convolutions, Self-KAGNtention, and Focal KAGN Modulation to reduce parameters while maintaining or improving performance. It systematically analyzes regularization, scaling (favoring width), and parameter-efficient finetuning (PEFT) across diverse datasets, including MNIST, CIFAR, Tiny ImageNet, ImageNet1k, and HAM10000, and extends KANs to segmentation via U-Net–like designs that achieve state-of-the-art results on medical datasets. Key findings show Gram polynomials as a strong basis, bottleneck layers dramatically cut parameters with minimal performance loss, and attention variants further boost accuracy in CV tasks. The work provides design principles and open-source code to advance reproducibility and practical deployment of KAN-based CV models.

Abstract

The emergence of Kolmogorov-Arnold Networks (KANs) has sparked significant interest and debate within the scientific community. This paper explores the application of KANs in the domain of computer vision (CV). We examine the convolutional version of KANs, considering various nonlinearity options beyond splines, such as Wavelet transforms and a range of polynomials. We propose a parameter-efficient design for Kolmogorov-Arnold convolutional layers and a parameter-efficient finetuning algorithm for pre-trained KAN models, as well as KAN convolutional versions of self-attention and focal modulation layers. We provide empirical evaluations conducted on MNIST, CIFAR10, CIFAR100, Tiny ImageNet, ImageNet1k, and HAM10000 datasets for image classification tasks. Additionally, we explore segmentation tasks, proposing U-Net-like architectures with KAN convolutions, and achieving state-of-the-art results on BUSI, GlaS, and CVC datasets. We summarized all of our findings in a preliminary design guide of KAN convolutional models for computer vision tasks. Furthermore, we investigate regularization techniques for KANs. All experimental code and implementations of convolutional layers and models, pre-trained on ImageNet1k weights are available on GitHub via this https://github.com/IvanDrokin/torch-conv-kan

Figures (12)

• Figure 1: KAN Convolution (left) and Bottleneck KAN Convolution (right). The main difference between these two types of layers is a encoder-decoder convolutional layers on the right data stream.
• Figure 2: Bottleneck Kolmogorov-Arnold Convolutional Mixture of Experts. The router and experts are placed between bottleneck convolutions, and each expert is a $\Tilde{\varphi}$ set of univariate functions. We use sparsely-gated mixture-of-experts shazeer2017outrageously.
• Figure 3: Possible dropout layer placement inside KAGN Convolution layer: Full - before the layer, Poly - before computing Gram basis, and Degree - before weighted sum of previously computed basis.
• Figure 4: Comparison of the number of trainable parameters for different types of layers. The reduction parameter for Bottleneck KAGN Conv 2D is responsible for the ratio of $d_x$ to $d_\varphi$
• Figure 4: BottleneckKAGN Convolutional networks, CIFAR 100