Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Training of Kolmogorov-Arnold Networks

Shairoz Sohail

TL;DR

It is found that (when judged by test accuracy) KANs are an effective alternative to MLP architectures on high-dimensional datasets and have somewhat better parameter efficiency, but suffer from more unstable training dynamics.

Abstract

Kolmogorov-Arnold Networks have recently been introduced as a flexible alternative to multi-layer Perceptron architectures. In this paper, we examine the training dynamics of different KAN architectures and compare them with corresponding MLP formulations. We train with a variety of different initialization schemes, optimizers, and learning rates, as well as utilize back propagation free approaches like the HSIC Bottleneck. We find that (when judged by test accuracy) KANs are an effective alternative to MLP architectures on high-dimensional datasets and have somewhat better parameter efficiency, but suffer from more unstable training dynamics. Finally, we provide recommendations for improving training stability of larger KAN models.

On Training of Kolmogorov-Arnold Networks

TL;DR

It is found that (when judged by test accuracy) KANs are an effective alternative to MLP architectures on high-dimensional datasets and have somewhat better parameter efficiency, but suffer from more unstable training dynamics.

Abstract

Kolmogorov-Arnold Networks have recently been introduced as a flexible alternative to multi-layer Perceptron architectures. In this paper, we examine the training dynamics of different KAN architectures and compare them with corresponding MLP formulations. We train with a variety of different initialization schemes, optimizers, and learning rates, as well as utilize back propagation free approaches like the HSIC Bottleneck. We find that (when judged by test accuracy) KANs are an effective alternative to MLP architectures on high-dimensional datasets and have somewhat better parameter efficiency, but suffer from more unstable training dynamics. Finally, we provide recommendations for improving training stability of larger KAN models.

Paper Structure

This paper contains 17 sections, 5 equations, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. B-splines
  4. The Kolmogorov-Arnold unit
  5. Method
  6. Architecture Choices
  7. Datasets
  8. Training Schemes
  9. Results
  10. Training with Backpropagation
  11. Training with the HSIC Bottleneck
  12. Additional Experiments
  13. The Role of Activation Functions
  14. KANs - Scaling with Degree
  15. Performance Observations
  16. ...and 2 more sections

Figures (6)

  • Figure 1: Test accuracy of KANs and MLPs utilizing backpropagation
  • Figure 2: Test accuracy of KANs and MLPs utilizing backpropagation
  • Figure 3: Test accuracy of KANs and MLPs utilizing the HSIC Bottleneck
  • Figure 4: The effect of activation function on KAN performance
  • Figure 5: The effect of B-spline degree and number of layers on KAN performance
  • ...and 1 more figures