Learning Neuron Dynamics within Deep Spiking Neural Networks
Eric Jahns, Davi Moreno, Michel A. Kinsy
TL;DR
This work tackles the expressivity bottleneck in deep spiking neural networks by introducing Learnable Neuron Models (LNMs), a general parametric framework that learns neuron dynamics during training. LNMs replace fixed neuron functions with a differentiable, low-degree polynomial f_theta(u), learned per layer, enabling richer temporal dynamics without sacrificing stability or hardware efficiency. The method employs a polynomial parameterization evaluated via Horner's method and optimized through Spatial-Temporal Backpropagation with surrogate gradients, achieving state-of-the-art results on CIFAR-10/100, ImageNet, and CIFAR-10 DVS, while incurring only a small energy overhead relative to LIF. The findings include diverse learned neuron behaviors across layers, confirming that a mixture of nonlinear dynamics enhances deep SNN performance and suggesting LNMs as a scalable path for energy-efficient, high-performing neuromorphic architectures.
Abstract
Spiking Neural Networks (SNNs) offer a promising energy-efficient alternative to Artificial Neural Networks (ANNs) by utilizing sparse and asynchronous processing through discrete spike-based computation. However, the performance of deep SNNs remains limited by their reliance on simple neuron models, such as the Leaky Integrate-and-Fire (LIF) model, which cannot capture rich temporal dynamics. While more expressive neuron models exist, they require careful manual tuning of hyperparameters and are difficult to scale effectively. This difficulty is evident in the lack of successful implementations of complex neuron models in high-performance deep SNNs. In this work, we address this limitation by introducing Learnable Neuron Models (LNMs). LNMs are a general, parametric formulation for non-linear integrate-and-fire dynamics that learn neuron dynamics during training. By learning neuron dynamics directly from data, LNMs enhance the performance of deep SNNs. We instantiate LNMs using low-degree polynomial parameterizations, enabling efficient and stable training. We demonstrate state-of-the-art performance in a variety of datasets, including CIFAR-10, CIFAR-100, ImageNet, and CIFAR-10 DVS. LNMs offer a promising path toward more scalable and high-performing spiking architectures.