Adaptive Gradient Learning for Spiking Neural Networks by Exploiting Membrane Potential Dynamics

TL;DR

This paper tackles gradient vanishing in surrogate-gradient training of spiking neural networks by linking surrogate-gradient width to evolving membrane potential dynamics. It introduces MPD-AGL, an adaptive gradient rule that modulates the SG width $\kappa$ over time based on MPD distribution, accounting for affine transformations from tdBN. The approach yields stronger gradient propagation, higher accuracy at ultra-low latency, and notable energy efficiency across CIFAR-10/100, CIFAR10-DVS, and Tiny-ImageNet, with theoretical support from derived MPD distributions. Overall, MPD-AGL provides a practical, architecture-friendly method to align gradient signals with neuronal dynamics, enhancing SNN performance in neuromorphic settings.

Abstract

Brain-inspired spiking neural networks (SNNs) are recognized as a promising avenue for achieving efficient, low-energy neuromorphic computing. Recent advancements have focused on directly training high-performance SNNs by estimating the approximate gradients of spiking activity through a continuous function with constant sharpness, known as surrogate gradient (SG) learning. However, as spikes propagate among neurons, the distribution of membrane potential dynamics (MPD) will deviate from the gradient-available interval of fixed SG, hindering SNNs from searching the optimal solution space. To maintain the stability of gradient flows, SG needs to align with evolving MPD. Here, we propose adaptive gradient learning for SNNs by exploiting MPD, namely MPD-AGL. It fully accounts for the underlying factors contributing to membrane potential shifts and establishes a dynamic association between SG and MPD at different timesteps to relax gradient estimation, which provides a new degree of freedom for SG learning. Experimental results demonstrate that our method achieves excellent performance at low latency. Moreover, it increases the proportion of neurons that fall into the gradient-available interval compared to fixed SG, effectively mitigating the gradient vanishing problem.

Figures (7)

• Figure 1: The overall framework of MPD-AGL. Pre-spikes are passed through the convolutional and normalization layers and then injected into spiking neurons to compute membrane potentials and fire spikes. The distribution of evolving MPD in forward propagation may not align with the fixed SG, leading to gradient vanishing or mismatch problems in backward propagation. Instead, the proposed adaptive gradient rule can synchronously adjust the width of SG to respond to evolving MPD during the entire timestep.
• Figure 2: The affine transformation of tdBN in an 8-layer vanilla SNN. Top line is the variation curves of parameters $\gamma$ and $\beta$ for the average of all channels in each layer. Bottom line is the variation curves of parameters $\gamma$ and $\beta$ for all channels in the first layer.
• Figure 3: The effectiveness of other SG functions.
• Figure 4: The average firing rate of each layer on CIFAR10 dataset.
• Figure 5: The comparison of different methods on the CIFAR10 dataset. (a) and (b) are the train loss and test accuracy, respectively. (c) and (f) are the proportion of neurons falling into the gradient-available interval in layer 7 and each layer of ResNet-19, respectively. (d) and (e) are the width of SG in each layer of MPD-AGL and LSG, respectively.

Theorems & Definitions (8)

• Theorem 1
• proof
• Theorem 2
• proof
• Theorem 1
• proof
• Theorem 2
• proof