PostHoc FREE Calibrating on Kolmogorov Arnold Networks

TL;DR

The paper tackles miscalibration in Kolmogorov-Arnold Networks (KANs), which use spline-based, edge-focused activations that can yield overconfident predictions in dense regions and underconfident ones in sparse areas. It introduces Temperature-Scaled Loss (TSL), a training-time objective that jointly optimizes the network parameters and a learnable temperature parameter $\tau$ to directly shape the predictive distribution, preserving strict propriety of the base loss. The authors provide theoretical guarantees (local convergence and reduction in calibration error) and empirical evidence across diverse vision benchmarks showing that TSL consistently reduces calibration error (ECE and variants) while maintaining competitive accuracy. They also analyze how KAN hyperparameters influence calibration and demonstrate that TSL mitigates grid-induced miscalibration without requiring post-hoc adjustments, offering practical guidance for spline-based networks and potential applicability to other architectures. The work contributes a principled, effective approach to calibration in flexible spline-based models, with implications for safety-critical and risk-sensitive applications.

Abstract

Kolmogorov Arnold Networks (KANs) are neural architectures inspired by the Kolmogorov Arnold representation theorem that leverage B Spline parameterizations for flexible, locally adaptive function approximation. Although KANs can capture complex nonlinearities beyond those modeled by standard MultiLayer Perceptrons (MLPs), they frequently exhibit miscalibrated confidence estimates manifesting as overconfidence in dense data regions and underconfidence in sparse areas. In this work, we systematically examine the impact of four critical hyperparameters including Layer Width, Grid Order, Shortcut Function, and Grid Range on the calibration of KANs. Furthermore, we introduce a novel TemperatureScaled Loss (TSL) that integrates a temperature parameter directly into the training objective, dynamically adjusting the predictive distribution during learning. Both theoretical analysis and extensive empirical evaluations on standard benchmarks demonstrate that TSL significantly reduces calibration errors, thereby improving the reliability of probabilistic predictions. Overall, our study provides actionable insights into the design of spline based neural networks and establishes TSL as a robust loss solution for enhancing calibration.

Figures (15)

• Figure 1: Logits distribution between the MLP and KAN models on the MNIST dataset with KAN producing a broader range of logits compared to the more centered logits of MLP.
• Figure 2: B-Spline Approximation in Dense vs. Sparse Regions. A B-spline (green) approximates a sine wave (orange, dashed), with $\mathbf{x}\in[0,1]$ subdivided into a dense region $\mathbf{x}\in[0,0.4]$ and a sparse region $\mathbf{x}\in[0.4,1.0]$ (shaded in gray). Over- or under-smoothing can arise from uneven grid usage.
• Figure 3: ECE vs. Spline Order on MNIST
• Figure 4: Visualization of temperature scaling applied to MLP (upward) and KAN (downward) logits for different temperature values. The first plot displays the original logits for both models. Each subsequent plot shows the probability distributions scaled by temperatures T=8.0,4.0,2.0,1.0,0.5, with the respective argmax classes highlighted. Gold/Red bars indicate the MLP/KAN argmax. Higher $T$ simulates noisier distributions, encouraging robustness to uncertainty. While Lower $T$ focuses on sharpening predictions, reducing overconfidence.
• Figure 5: Reliability Diagram for KAN Model: Evaluation of calibration performance for a KAN model trained with 8 SOTA loss functions on the MNIST dataset under identical hyperparameter settings.

Theorems & Definitions (15)

• Definition 3.1: Calibration Error, guo2017calibrationnaeini2015obtaining
• Definition 3.2: Expected Calibration Error
• Definition 3.3: Smooth ECE, blasiok2023smooth
• Proposition 3.4: Spline Order and Calibration Error
• Proposition 5.1
• Lemma 5.2: Monotonic Gradient Updates
• Theorem 5.3: Local Convergence and Calibration Improvement ghadimi2013stochastic
• Corollary 5.4: Reduction of ECE
• Proposition C.1: Strict Properness of TSL
• Lemma D.1: Monotonic Gradient Updates