Sign Gradient Descent-based Neuronal Dynamics: ANN-to-SNN Conversion Beyond ReLU Network

TL;DR

The paper tackles the limited nonlinearity support and high latency in ANN-to-SNN conversion by introducing a sign gradient-descent (signGD) based neuronal dynamics that links discrete spiking behavior to optimization updates. It proves that IF/LIF neurons with rate or EMA coding emulate subgradient methods on a crafted objective, then extends this to design signGD-based neurons that can approximate unary and multi-operand nonlinearities beyond ReLU, including max pooling and layer normalization. Empirically, the approach enables state-of-the-art ANN-to-SNN conversion on large datasets and supports contemporary architectures such as ConvNext, MLP-Mixer, and ResMLP with competitive latency (e.g., 64–256 time-steps on ImageNet). The findings offer a practical, theory-grounded path to energy-efficient SNN inference without sacrificing accuracy, and point to broader implications for neuromorphic hardware and neuroscience-inspired computation. Overall, signGD-based neuronal dynamics broaden the scope and speed of SNN-based inference while preserving the core benefits of spike-based computation.”

Abstract

Spiking neural network (SNN) is studied in multidisciplinary domains to (i) enable order-of-magnitudes energy-efficient AI inference and (ii) computationally simulate neuro-scientific mechanisms. The lack of discrete theory obstructs the practical application of SNN by limiting its performance and nonlinearity support. We present a new optimization-theoretic perspective of the discrete dynamics of spiking neurons. We prove that a discrete dynamical system of simple integrate-and-fire models approximates the sub-gradient method over unconstrained optimization problems. We practically extend our theory to introduce a novel sign gradient descent (signGD)-based neuronal dynamics that can (i) approximate diverse nonlinearities beyond ReLU and (ii) advance ANN-to-SNN conversion performance in low time steps. Experiments on large-scale datasets show that our technique achieves (i) state-of-the-art performance in ANN-to-SNN conversion and (ii) is the first to convert new DNN architectures, e.g., ConvNext, MLP-Mixer, and ResMLP. We publicly share our source code at https://github.com/snuhcs/snn_signgd .

Figures (13)

• Figure 1: In this paper, we (i) mathematically connect the neuronal dynamics of integrate-and-fire models with the optimization dynamics of subgradient method, (ii) extend the theory to design a new spiking neuron model that can approximate arbitrary element-wise tensor operators, and (iii) use our neuron model to expand ANN-to-SNN conversion beyond ReLU networks (Fig. \ref{['fig:practical_contribution']}).
• Figure 2: Mathematical equivalence of discrete neuronal dynamics of IF neuron (left) and subgradient method over an unconstrained convex optimization problem (right), described in Theorem \ref{['thm:if_rate']}.
• Figure 3: Our interpretation of SNN's computational characteristics: neuronal dynamics (\ref{['fig:dynamics_stages']}-\ref{['fig:dynamics_interpretation']}), spike train (\ref{['fig:spike_train_interpretation']}) .
• Figure 4: sign gradient descent(signGD)-based neuronal dynamics, a design optimization-theoretically extended from Fig. \ref{['fig:dynamics_interpretation']}.
• Figure 5: An example of decomposing max pooling of 4x4 window with binary-input maximum operators.

Theorems & Definitions (37)

• Theorem 4.1
• Theorem 4.2
• Theorem 4.3
• Corollary 4.4
• Definition 5.1
• Definition 5.2
• Theorem 5.3
• Corollary 5.4
• Corollary 5.5
• Corollary 5.6