KAN-Mixers: a new deep learning architecture for image classification

TL;DR

This work addresses the challenge of achieving refined feature extraction in mixer-based image classification by integrating Kolmogorov-Arnold Networks (KAN) into a MLP-Mixer–style architecture, creating KAN-Mixers. The proposed model uses patch embeddings, Mixer Blocks with KAN-based token and channel mixers, and an adaptive pooling classifier, with a hyperparameter-optimized, fully KAN-based design implemented efficiently. On Fashion-MNIST, KAN-Mixers reach an average accuracy of $0.9030$, and on CIFAR-10 they achieve $0.6980$, outperforming the MLP, MLP-Mixer, and KAN baselines. Wilcoxon tests reveal equivalence at $p=0.05$ and superiority at $p=0.10$, supporting the competitive edge of KAN-Mixers, though training time remains a consideration. Overall, the results suggest that KAN layers can enhance feature extraction and interpretability in mixer architectures, offering a viable alternative when training time is not critically constrained.

Abstract

Due to their effective performance, Convolutional Neural Network (CNN) and Vision Transformer (ViT) architectures have become the standard for solving computer vision tasks. Such architectures require large data sets and rely on convolution and self-attention operations. In 2021, MLP-Mixer emerged, an architecture that relies only on Multilayer Perceptron (MLP) and achieves extremely competitive results when compared to CNNs and ViTs. Despite its good performance in computer vision tasks, the MLP-Mixer architecture may not be suitable for refined feature extraction in images. Recently, the Kolmogorov-Arnold Network (KAN) was proposed as a promising alternative to MLP models. KANs promise to improve accuracy and interpretability when compared to MLPs. Therefore, the present work aims to design a new mixer-based architecture, called KAN-Mixers, using KANs as main layers and evaluate its performance, in terms of several performance metrics, in the image classification task. As main results obtained, the KAN-Mixers model was superior to the MLP, MLP-Mixer and KAN models in the Fashion-MNIST and CIFAR-10 datasets, with 0.9030 and 0.6980 of average accuracy, respectively.

Figures (6)

• Figure 1: Examples from the Fashion-MNIST dataset.
• Figure 2: Examples from the CIFAR-10 dataset.
• Figure 3: KAN-Mixers Architecture.
• Figure 4: Proposed methodology.
• Figure 5: Violin and Strip plots of model training on the Fashion-MNIST set.