Catastrophic Forgetting in Kolmogorov-Arnold Networks

TL;DR

The paper investigates catastrophic forgetting in Kolmogorov-Arnold Networks (KANs) and presents a theoretical framework that connects forgetting to activation support overlap and intrinsic data dimension. It validates the theory with synthetic and image-classification experiments and introduces KAN-LoRA, a KAN-based adapter for continual knowledge editing in language models, demonstrating improved retention over MLP-based adapters in many settings. Key findings show that KANs exhibit strong retention in low-dimensional, structured tasks but are vulnerable in high-dimensional domains such as vision and language, with forgetting scaling with overlap and exponential with intrinsic dimension. The work provides practical guidance for designing continual learning systems and offers a pathway to memory-aware model editing using KAN-based adapters.

Abstract

Catastrophic forgetting is a longstanding challenge in continual learning, where models lose knowledge from earlier tasks when learning new ones. While various mitigation strategies have been proposed for Multi-Layer Perceptrons (MLPs), recent architectural advances like Kolmogorov-Arnold Networks (KANs) have been suggested to offer intrinsic resistance to forgetting by leveraging localized spline-based activations. However, the practical behavior of KANs under continual learning remains unclear, and their limitations are not well understood. To address this, we present a comprehensive study of catastrophic forgetting in KANs and develop a theoretical framework that links forgetting to activation support overlap and intrinsic data dimension. We validate these analyses through systematic experiments on synthetic and vision tasks, measuring forgetting dynamics under varying model configurations and data complexity. Further, we introduce KAN-LoRA, a novel adapter design for parameter-efficient continual fine-tuning of language models, and evaluate its effectiveness in knowledge editing tasks. Our findings reveal that while KANs exhibit promising retention in low-dimensional algorithmic settings, they remain vulnerable to forgetting in high-dimensional domains such as image classification and language modeling. These results advance the understanding of KANs' strengths and limitations, offering practical insights for continual learning system design.

Figures (5)

• Figure 1: MSE loss in logarithmic scale for five different binary addition tasks during training on one's addition task.
• Figure 2: MSE loss during sequential training on five different decimal addition tasks, from one's to five's addition facts.
• Figure 3: Task 1 accuracy after sequential training on task 1 and 2 from CIFAR-10, comparing (a) KAN-Transformer and (b) MLP-Transformer. Model configuration is labeled as (#classification layers – #encoder blocks – #attention heads).
• Figure 4: Average accuracy on previously learned tasks after training on 2 to 5 tasks with varying sample sizes from CIFAR-10 and Tiny-ImageNet datasets. Sub-figures show results for (a) KAN-Transformer and (b) MLP-Transformer.
• Figure 5: Five sequential tasks were respectively constructed for MNIST (\ref{['fig:mn1']}-\ref{['fig:mn5']}), CIFAR-10 (\ref{['fig:ct1']}-\ref{['fig:ct5']}) and Tiny-ImageNet (\ref{['fig:it1']}-\ref{['fig:it5']}) datasets, where each task containing two distinct classes.

Theorems & Definitions (8)

• Lemma 1: Zero‐Overlap Retention
• Theorem 1: Retention Bound via Overlap
• Theorem 2: Branch‐wise Cumulative Forgetting
• Corollary 1: Expected Forgetting under Random Supports
• Corollary 2: Saturation via Union‐Bound
• Theorem 3: Intrinsic‐Dimension Forgetting Rate
• Corollary 3: Retention for Low‐Dimensional Tasks
• Corollary 4: Fragmentation Mitigates Complexity