Kolmogorov-Arnold Network Autoencoders

TL;DR

The paper investigates Kolmogorov-Arnold Networks (KANs) as edge-activated alternatives to MLPs for autoencoding tasks. It formalizes KANs using the Kolmogorov-Arnold representation $f(x_1, x_2, \dots, x_n) = \sum_{i=1}^{2n-1} \phi_i \left( \sum_{j=1}^n \psi_{ij}(x_j) \right)$ with spline-based univariate functions and details the associated parameter count $N_a$ and $(G+K)$ spline coefficients. AE-KAN models are evaluated on MNIST, SVHN, and CIFAR-10, showing competitive reconstruction errors and improved latent-space discrimination when assessed with a KNN classifier on the learned representations. The study notes higher capacity and training cost for KANs and discusses interpretability, suggesting future work on real-world applications and hybrid architectures, along with formal interpretability metrics.

Abstract

Deep learning models have revolutionized various domains, with Multi-Layer Perceptrons (MLPs) being a cornerstone for tasks like data regression and image classification. However, a recent study has introduced Kolmogorov-Arnold Networks (KANs) as promising alternatives to MLPs, leveraging activation functions placed on edges rather than nodes. This structural shift aligns KANs closely with the Kolmogorov-Arnold representation theorem, potentially enhancing both model accuracy and interpretability. In this study, we explore the efficacy of KANs in the context of data representation via autoencoders, comparing their performance with traditional Convolutional Neural Networks (CNNs) on the MNIST, SVHN, and CIFAR-10 datasets. Our results demonstrate that KAN-based autoencoders achieve competitive performance in terms of reconstruction accuracy, thereby suggesting their viability as effective tools in data analysis tasks.

Figures (5)

• Figure 1: KAN Autoencoder Structure. The structure of our KAN autoencoder consists of an encoder and a decoder. The encoder includes a KAN layer, a ReLU activation, and a dense layer, which transforms the input size to a hidden size and then to the bottleneck size. For example, it maps from 784 to 8, followed by a ReLU activation, and then from 8 to 18. The decoder reverses this process, starting with a dense layer, followed by a ReLU activation, and finally a KAN layer, mapping from the bottleneck size back to the hidden size and the original input size, i.e., from 18 to 8, followed by ReLU, and from 8 to 784.
• Figure 2: MSE reconstruction loss versus bottleneck size for the KAN autoencoder across three datasets: a) MNIST, b) SVHN, and c) CIFAR-10. Let $\text{hidden\_size} = \text{bottleneck\_size}$ in Eqs. \ref{['eq:enc']} and \ref{['eq:dec']}. For the MNIST dataset, a bottleneck size of 150 achieves excellent performance, while 50 provides good performance. In the case of CIFAR-10, a bottleneck size of 500 yields excellent performance, with 200 being very good. For SVHN, a bottleneck size of 500 is excellent, and 200 shows good reconstruction.
• Figure 3: Reconstruction of Compressed Images from the CIFAR Dataset with Different Bottleneck Sizes. Let $\text{hidden\_size} = \text{bottleneck\_size}$ in Eqs. \ref{['eq:enc']} and \ref{['eq:dec']}. a) Original samples b) Reconstructed samples with $\text{bottleneck\_size} = 50$, c) Reconstructed samples with $\text{bottleneck\_size} = 150$, and d) Reconstructed samples with $\text{bottleneck\_size} = 500$.
• Figure 4: KAN Autoencoders Image Reconstruction Results: a) MNIST Dataset ($\text{bottleneck\_size} = 8$) b) SVHN Dataset ($\text{bottleneck\_size} = 64$) c) CIFAR-10 Dataset ($\text{bottleneck\_size} = 64$).
• Figure 5: KNN Classification Results Given the Latent Representations of the KAN Autoencoders: a) MNIST Dataset ($\text{bottleneck\_size} = 8$) b) SVHN Dataset ($\text{bottleneck\_size} = 64$) c) CIFAR-10 Dataset ($\text{bottleneck\_size} = 64$).