MetaCluster: Enabling Deep Compression of Kolmogorov-Arnold Network

TL;DR

KANs incur a memory overhead because each edge carries a coefficient vector of length $|\mathbf{w}|$, multiplying parameter counts relative to MLPs. MetaCluster trains a lightweight meta-learner that maps low-dimensional embeddings $\mathbf{z}$ to per-edge vectors $M_\theta(\mathbf{z})$ and shapes them onto a low-dimensional manifold suitable for clustering; after K-means in coefficient space, per-edge vectors are replaced by shared centroids and the meta-learner is discarded, followed by brief centroid fine-tuning. Across MNIST, CIFAR-10, and CIFAR-100 with fully-connected KANs and ConvKANs, MetaCluster achieves up to about $80\times$ storage reduction without accuracy loss, outperforming naive clustering and preserving performance across multiple basis functions. This work demonstrates that learning a task-aligned manifold for high-dimensional activations enables aggressive weight-sharing in KANs, enabling scalable deployment.

Abstract

Kolmogorov-Arnold Networks (KANs) replace scalar weights with per-edge vectors of basis coefficients, thereby boosting expressivity and accuracy but at the same time resulting in a multiplicative increase in parameters and memory. We propose MetaCluster, a framework that makes KANs highly compressible without sacrificing accuracy. Specifically, a lightweight meta-learner, trained jointly with the KAN, is used to map low-dimensional embedding to coefficient vectors, shaping them to lie on a low-dimensional manifold that is amenable to clustering. We then run K-means in coefficient space and replace per-edge vectors with shared centroids. Afterwards, the meta-learner can be discarded, and a brief fine-tuning of the centroid codebook recovers any residual accuracy loss. The resulting model stores only a small codebook and per-edge indices, exploiting the vector nature of KAN parameters to amortize storage across multiple coefficients. On MNIST, CIFAR-10, and CIFAR-100, across standard KANs and ConvKANs using multiple basis functions, MetaCluster achieves a reduction of up to 80$\times$ in parameter storage, with no loss in accuracy. Code will be released upon publication.

Figures (2)

• Figure 1: T-SNE visualization of KAN activation weights with and without the meta-learner $M_\theta$. (a) With a 1D embedding, the MetaKAN learns a nearly one-dimensional manifold, (b) with a 2D embedding, it learns a structured two-dimensional manifold, while (c) the baseline KAN without a meta-learner fails to organize the weights into a coherent low-dimensional structure.
• Figure 2: Classification accuracy vs. number of clusters for FastKAN model variants.