Learning Nonlinear Heterogeneity in Physical Kolmogorov-Arnold Networks

TL;DR

This work demonstrates that learning programmable nonlinear synaptic dynamics via Kolmogorov-Arnold Networks (KAN) implemented with Synaptic Nonlinear Elements (SYNEs) enables compact, energy-efficient physical neural networks that outperform equivalently parameterised software MLPs. By replacing fixed, simple neuron nonlinearities with learnable, per-synapse nonlinear functions, the authors realize substantial reductions in network size and device count while maintaining or enhancing task performance on nonlinear regression, classification, and real-world Li-Ion battery aging prediction. A differentiable digital-twin framework enables hardware-in-the-loop training, and the study introduces an epsilon expressivity metric that correlates strongly with function-representation performance, offering a practical design tool. The results—coupled with energy projections and switched-capacitor amplifier schemes—underscore the potential of learned physical nonlinearities as a hardware-native primitive for scalable, efficient learning systems built on mature SOI technology.

Abstract

Physical neural networks typically train linear synaptic weights while treating device nonlinearities as fixed. We show the opposite - by training the synaptic nonlinearity itself, as in Kolmogorov-Arnold Network (KAN) architectures, we yield markedly higher task performance per physical resource and improved performance-parameter scaling than conventional linear weight-based networks, demonstrating ability of KAN topologies to exploit reconfigurable nonlinear physical dynamics. We experimentally realise physical KANs in silicon-on-insulator devices we term 'Synaptic Nonlinear Elements' (SYNEs), operating at room temperature, microampere currents, 2 MHz speeds and ~250 fJ per nonlinear operation, with no observed degradation over 10^13 measurements and months-long timescales. We demonstrate nonlinear function regression, classification, and prediction of Li-Ion battery dynamics from noisy real-world multi-sensor data. Physical KANs outperform equivalently-parameterised software multilayer perceptron networks across all tasks, with up to two orders of magnitude fewer parameters, and two orders of magnitude fewer devices than linear weight based physical networks. These results establish learned physical nonlinearity as a hardware-native computational primitive for compact and efficient learning systems, and SYNE devices as effective substrates for heterogenous nonlinear computing.

Figures (10)

• Figure 1: Kolmogorov-Arnold Networks and 'Synaptic Nonlinear Element' (SYNE) Devices.a) Comparison of multilayer perceptron (MLP) and Kolmogorov-Arnold network (KAN). MLPs employ relatively simple, identical nonlinear functions on all neurons (often ReLU), and learned linear weights on synapses/edges. KANs employ linear neurons that simply sum inputs, and complex synapses with learnable nonlinear functions. b) Schematic of how a single KAN synapse is constructed by combining multiple physical SYNE devices in parallel. The input signal from a neuron is sent to $n$ SYNEs (here 2), each with learnable parameters which define the nonlinear function: two $V_\mathrm{Tune}$ control voltages which define the shape of the nonlinear $I-V$ response, input range scaling, and output gain $G$. The desired physical KAN synapse output is then given by a linear sum of the $n$ SYNE outputs, with a single bias term - here trained to produce a sine wave. c) Fabrication process of SYNE devices. P-dopants are introduced to a 30 nm silicon-on-insulator layer via polymer-graft doping, then a 2 nm SiO$_2$ oxide layer introduced via SC2 cleaning. Micron-scale disks are defined via maskless optical lithography then wet KOH etch. Al contacts are added via maskless optical lithography then thermal evaporation. d) SYNE device schematic and scanning electron micrograph. While eight contacts are patterned, only four are required for the scheme employed here: a voltage input $V_\mathrm{In}$, a current output $I_\mathrm{Out}$, and two tunable control voltages $V_\mathrm{Tune}$ (adjacent to the current output for maximal nonlinearity control). The remaining contacts are unused, providing scope for future interconnectivity and exploring additional control voltages. e) SYNE $I-V$ traces measured across a range of control voltages. $V_\mathrm{Tune 1}$ is held constant at -0.2 V, $V_\mathrm{Tune 2}$ is swept 1.4 V to -2.9 V. f) SYNEs provide a broad, reconfigurable range of nonlinear $I-V$ responses including negative differential resistance - demonstrating the high expressivity afforded by a single SYNE device. Each row sweeps only a single control voltage, with the bottom row leaving the second control voltage fixed at 0 V to highlight the range of nonlinear dynamics accessible with just a single control.
• Figure 2: Learning Nonlinear Functions with Physical KAN Synapsesa) Nonlinear functions are learned in-silico and experimentally realised using a single physical KAN synapse. Using a differentiable digital twin model of a SYNE device, backpropagation is used to learn the experimental parameters required to experimentally reproduce arbitrary functions via SYNE output currents. The blue/cyan traces on the second and fifth column plots show the output currents of each SYNE in the synapse. The orange trace in the third and sixth columns is the experimentally-measured output of the KAN synapse, and the blue trace the ground truth target. In this work, a single SYNE device is employed - measured sequentially with different $V_\mathrm{Tune}$ values to implement a larger network, with output gain and linear summation performed digitally off-chip. Using only two SYNEs, simple ReLU and sigmoid functions are implemented at high accuracy. For functions with higher harmonic content such as cos$(2 \pi x)$, a greater number of SYNE devices per synapse can be employed to improve accuracy - demonstrated here by two SYNEs which implement a lower-accuracy representation, and eight SYNEs achieving a higher accuracy. b) A broad range of functions are experimentally accessible using a single KAN synapse containing 6 SYNE devices, measured at 2 MHz per datapoint with 10k datapoints per function. c) Two KAN synapses are connected in series, experimentally realising complex nested $f(g(x))$ functions. Series interconnection enables more expressive synapses. d,e) MSE vs. network size/trainable parameters averaged over multiple nonlinear functions, comparing experimental and simulated physical KAN synapses, and MLPs. For a single KAN synapse (d), performance is averaged over 13 functions, including the 8 functions in a) (details in SI). Here, software MLPs outperform experimental physical KANs, although simulated (digital twin) physical KANs beat MLPs. For two series KAN synapses (e), performance is averaged over the 6 functions shown in b). Here, experimental and simulated physical KANs beat MLPs due to the enhanced expressivity of series-connected synapses.
• Figure 3: Expressivity via Epsilon Packing is Strongly Correlated with Function Representation Performance.a) Physical KAN synapse expressivity is quantified for single and two series-connected synapses comprising 1-8 SYNEs, via Epsilon Packing. 2000 curves are generated for each synapse by sweeping all SYNE parameters, and embedded in a 32-dimensional PCA space. The packing number denotes how many curves are at least a distance $\epsilon$ apart from their nearest neighbour in PCA space, e.g. how varied/expressive is the set of possible curves. The packing number at a given $\epsilon$ increases as more SYNEs are added to a synapse. Moving to two series synapses substantially increases expressivity, explaining why physical KANs move from underperforming relative to MLPs for a single synapse, to beating them for two series synapses. The faint horizontal line denotes the $\epsilon$ distance at which 50% of total curves can be packed. b) To allow us to quantitatively assess the link between expressivity metricised via epsilon-packing, we extract a single value from each of the traces in (a) - the packing distance $\epsilon$ at which 50% of curves can be packed. We plot this here against parameter count, showing a strong positive correlation (power law fit) with a steeper gradient and higher absolute expressivity for the two series synapse case. c) We now explore correlation between the expressivity (via the 50% packing $\epsilon$ distance) and the mean learnt function representation MSE performance across a set of ten complex nonlinear functions (methods). The expressivity is strongly correlated with the MSE performance, with a steeper gradient in the two series synapse case. This result highlights the efficacy of using expressivity, quantified simply and computationally cheaply via epsilon packing in PCA space, as a metric for assessing the power of a given device or system for learning diverse ranges of complex nonlinear dynamics.
• Figure 4: Learning Multivariate Nonlinear Functions with Physical KANs. Better performance and smaller network sizes are found relative to software multilayer perceptrons, with interpretability from learned synaptic shapes. a) We implement a set of 2D $xy$ regression tasks on nonlinear functions with the form $f(g(x)+h(y))$ where $f(x), g(x), h(y)$ are unknown independent functions. A training set is composed of 80 $\times$ 80 points from -1 to 1 in $x$ and $y$, and a [2,1,1] physical KAN network with 12 SYNE devices per synapse is trained via backpropagation on the digital twin model to reproduce the function output for 800 epochs. The physical KAN network has no prior knowledge of the function composition, but on inspection of the learned synaptic shapes has learned to approximate the underlying function forms of the three separate $f,g,h$ functions - aiding symbolic interpretability. The learned parameters are then experimentally transferred to the hardware SYNE devices, and evaluated on a test set comprising 100 $\times$ 100 previously unseen $x,y$ points. The 'pKAN' (physical KAN) heatmap shows the experimental test set, with an MSE of 1.37e-02 and good reproduction of function features including high-frequency components at the edges and corners. Performance is compared between physical KANs (experimental) and software multilayer perceptron networks at a range of network sizes/trainable parameter counts. Physical KAN network sizes are reduced by selectively pruning SYNE devices, achieving better performance than equivalently parameterised multilayer perceptrons which require 34$\times$ more parameters than physical KANs to achieve equal performance. b) Four more functions are learned and experimentally tested on physical KANs. In each case, physical KANs outperform MLPs at higher parameter counts, with approx. 30$\times$ reduction in network size relative to equivalently-performing multilayer perceptrons. Learned synaptic shapes aid interpretability of the underlying functions and work well in most cases, but are not without errors such as the imperfect cos($\pi x$) in the final layer synapse in the bottom left network plot.
• Figure 5: Binary Classification and Prediction of Li-Ion Battery Dynamics on Real-World Multi-Sensor Data.a) Binary classification on nonlinearly-separable synthetic datasets. Here, physical KANs are trained on a set of 8k $x,y$ points ($x,y: -1,1$) which belong to one of two classes 'Red' or 'Blue', sampled from a variety of nonlinearly separable distribution shapes. The learned KAN parameters/synaptic shapes are then transferred to physical SYNE hardware, and performance evaluated on a test set of 10k points. The top-left panel shows experimental test performance on a 'yin-yang' dataset with 99.0% accuracy using a small [2,2,1] KAN with 8 SYNE devices per synapse. Learned synaptic shapes are shown on a schematic of the network. Classification performance vs. network size is evaluated for a range of physical KAN architectures relative to multilayer perceptrons of 1-4 hidden layers. Physical KANs outperform equivalently parameterised multilayer perceptrons, with up to 90$\times$ reduction in trainable parameter count relative to equally performing multilayer perceptrons. b) Five classification tasks with increasing difficulty left-to-right. Physical KANs outperform equivalently-parameterised multilayer perceptrons - with over 2 orders of magnitude reduction in trainable parameter count relative to multilayer perceptrons for four tasks. The 2-arm spiral accuracy is dominated by points located close to the fuzzy boundary between red and blue arms at the spiral centre, hence the lower accuracy even though the overall task and decision boundary is easier relative to other tasks. c) Prediction of Li-Ion battery dynamics and time to end of battery life (EOL) from noisy real-world multi-sensor data (NASA battery dataset). The other tasks considered in this study are ideal numerical functions or synthetically generated datasets, where the function-approximation benefits of the KAN are well suited. Here, we examine physical KAN performance on a real-world multi-sensor dataset recorded from noisy physical systems, a large set of Li-Ion batteries undergoing multiple charge/discharge cycles. The underlying charge dynamics can be described by physical equations, but the real-world behaviour of battery devices is non-ideal with experimental noise, differences between individual batteries, sensor and environmental drift, and other non-idealities. The task is to predict the time to end of life of a given unseen battery from sensor data, with no prior knowledge of how many charge/discharge cycles the battery has undergone. Here, physical KAN networks ([9,12,1], 12 SYNEs/synapse) outperform equivalently-parameterised multilayer perceptrons, demonstrating that SYNE-based physical KANs are not restricted to performing well on synthetic or numerically ideal tasks/datasets. The central plot shows experimental performance on the test set, plotting KAN predicted vs. true time to end of battery life for 10 batteries over 547 discharge cycles, unseen during training. We compare performance vs. network size for physical KANs against multilayer perceptrons, showing that physical KANs are able to perform well at compact network sizes on real-world tasks, requiring 39$\times$ fewer parameters than equally-performing MLPs.