Optimization, Generalization and Differential Privacy Bounds for Gradient Descent on Kolmogorov-Arnold Networks

TL;DR

This paper studies gradient descent on two-layer Kolmogorov–Arnol'd Networks (KANs), deriving a unified framework that bounds optimization, generalization, and differential privacy (DP) performance. Under an NTK-separable logistic loss, GD achieves an optimization rate of $O(1/T)$ and a generalization rate of $O(1/n)$ with polylogarithmic width, while DP-GD yields a privacy-utility bound of $ ilde{O}( ext{sqrt}(d)/(n\epsilon))$ and requires a polylog width for meaningful privacy guarantees. The results reveal a qualitative gap between non-private and private training regimes in terms of width sufficiency, and they provide practical guidance on width and early stopping for KANs. Empirical validation on synthetic data and MNIST corroborates the theoretical predictions and demonstrates how width and iteration choices impact both private and non-private performance.

Abstract

Kolmogorov--Arnold Networks (KANs) have recently emerged as a structured alternative to standard MLPs, yet a principled theory for their training dynamics, generalization, and privacy properties remains limited. In this paper, we analyze gradient descent (GD) for training two-layer KANs and derive general bounds that characterize their training dynamics, generalization, and utility under differential privacy (DP). As a concrete instantiation, we specialize our analysis to logistic loss under an NTK-separable assumption, where we show that polylogarithmic network width suffices for GD to achieve an optimization rate of order $1/T$ and a generalization rate of order $1/n$, with $T$ denoting the number of GD iterations and $n$ the sample size. In the private setting, we characterize the noise required for $(ε,δ)$-DP and obtain a utility bound of order $\sqrt{d}/(nε)$ (with $d$ the input dimension), matching the classical lower bound for general convex Lipschitz problems. Our results imply that polylogarithmic width is not only sufficient but also necessary under differential privacy, revealing a qualitative gap between non-private (sufficiency only) and private (necessity also emerges) training regimes. Experiments further illustrate how these theoretical insights can guide practical choices, including network width selection and early stopping.

Figures (3)

• Figure 1: Comparison of MLP and KAN performance on genomic sequence classification. Each point pair corresponds to a benchmark dataset from Table 1 of CherednichenkoP25, reporting Matthews correlation coefficient (higher is better) for a baseline MLP model and its KAN-based counterpart. Most KAN points lie to the right of their MLP counterparts, indicating improved predictive performance when replacing MLP modules with KAN layers.
• Figure 2: (a) Training and test accuracy as a function of the width $m$ (with $T$ fixed) and the number of iterations $T$ (with $m$ fixed) for GD, and (b) private utility as a function of $m$ and $T$ for DP-GD, on synthetic logistic data (top row) and MNIST (bottom row). In (a), the vertical dashed lines indicate empirically observed change points beyond which increasing the width $m$ yields diminishing or flat training and test accuracy, while increasing the number of training iterations $T$ primarily improves training accuracy but leads to diminishing or flat test accuracy, consistent with the theoretical bounds. In (b), the dashed lines indicate observed turning points beyond which private utility degrades due to the amplification or accumulation of privacy noise, in line with the private utility bound. Together, these results suggest selecting width and training duration within the admissible regimes identified by the theory.
• Figure 3: Training and test losses versus $m$ and $T$.

Theorems & Definitions (67)

• Theorem 3.1: Informal version of Theorem \ref{['thm:ntk']}
• Theorem 3.2: Informal version of Theorem \ref{['thm:ntk-gen']}
• Theorem 3.3: Informal version of Theorem \ref{['thm:dp-risk-ntk']}
• Theorem 4.3: Optimization -- General bound
• Remark 4.4: Key proof idea
• Remark 4.6: Milder condition
• Theorem 4.7: Optimization under Realizability
• Theorem 4.9: Optimization under NTK separability
• Theorem 4.10: Generalization gap via on-average argument stability
• Theorem 4.11: Generalization -- General bound