ILIF: Temporal Inhibitory Leaky Integrate-and-Fire Neuron for Overactivation in Spiking Neural Networks

TL;DR

This work tackles the gamma dilemma in surrogate-gradient training for spiking neural networks: large SG support width $\gamma$ triggers overactivation and energy waste, while small $\gamma$ causes gradient vanishing. It introduces the Temporal Inhibitory Leaky Integrate-and-Fire (ILIF) neuron with two inhibitory units, MPIU and CIU, to suppress overactivation and preserve gradient flow via temporal shortcuts. Theoretical analysis links $\gamma$, weight norms, firing rates, and gradient propagation, and extensive experiments on CIFAR-10/100 and neuromorphic DVSCIFAR/DVSGesture show ILIF achieving higher accuracy, lower firing rates, and improved energy efficiency compared with LIF and related approaches. Overall, ILIF provides a biologically inspired, practical enhancement for energy-efficient neuromorphic computing with backpropagation through time.

Abstract

The Spiking Neural Network (SNN) has drawn increasing attention for its energy-efficient, event-driven processing and biological plausibility. To train SNNs via backpropagation, surrogate gradients are used to approximate the non-differentiable spike function, but they only maintain nonzero derivatives within a narrow range of membrane potentials near the firing threshold, referred to as the surrogate gradient support width gamma. We identify a major challenge, termed the dilemma of gamma: a relatively large gamma leads to overactivation, characterized by excessive neuron firing, which in turn increases energy consumption, whereas a small gamma causes vanishing gradients and weakens temporal dependencies. To address this, we propose a temporal Inhibitory Leaky Integrate-and-Fire (ILIF) neuron model, inspired by biological inhibitory mechanisms. This model incorporates interconnected inhibitory units for membrane potential and current, effectively mitigating overactivation while preserving gradient propagation. Theoretical analysis demonstrates ILIF effectiveness in overcoming the gamma dilemma, and extensive experiments on multiple datasets show that ILIF improves energy efficiency by reducing firing rates, stabilizes training, and enhances accuracy. The code is available at github.com/kaisun1/ILIF.

Figures (7)

• Figure 1: Effect of SG support width ($\gamma$) on network performance: (a) Surrogate gradient method. (b) Changes in firing rate and accuracy with $\gamma$. (c) Average layer weight norm variation with $\gamma$.
• Figure 2: (a) Diagram of the inhibition mechanism. (b) Structure of the vanilla LIF model. (c) Internal operations of the vanilla LIF model. (d) Structure of the ILIF model. (e) Internal operations of the ILIF model.
• Figure 3: Continuous firing rate comparison on neuromorphic data.
• Figure 4: Comparison of firing rates between LIF and ILIF in Layers 5, 7, 9, and 11 on CIFAR10. Blue pixels indicate LIF spikes, red pixels indicate ILIF spikes, and green pixels indicate simultaneous spikes. Spike counts are displayed beside each time step.
• Figure 5: Firing rate and accuracy comparison for LIF and ILIF with respect to $\gamma$ on CIFAR10

Theorems & Definitions (11)

• Lemma 1
• proof
• Lemma 2
• Theorem 1
• proof
• Theorem 2
• proof
• Theorem 3
• proof
• Lemma 2