CLIF: Complementary Leaky Integrate-and-Fire Neuron for Spiking Neural Networks

TL;DR

This work addresses the difficulty of training spiking neural networks (SNNs) due to non-differentiable spikes and temporal gradient vanishing in Leaky Integrate-and-Fire (LIF) neurons. It proposes the Complementary LIF (CLIF) neuron, introducing a zero-hyperparameter complementary membrane potential that creates additional backpropagation paths for temporal gradients while preserving binary outputs. Theoretical analysis and empirical results show that CLIF enriches temporal gradient terms, reduces gradient vanishing, and improves performance across static image and neuromorphic datasets, often matching or exceeding ANN performance with the same architecture and training regime. The approach maintains energy efficiency and generalizes across backbones, offering a practical, interchangeable replacement for LIF in SNNs.

Abstract

Spiking neural networks (SNNs) are promising brain-inspired energy-efficient models. Compared to conventional deep Artificial Neural Networks (ANNs), SNNs exhibit superior efficiency and capability to process temporal information. However, it remains a challenge to train SNNs due to their undifferentiable spiking mechanism. The surrogate gradients method is commonly used to train SNNs, but often comes with an accuracy disadvantage over ANNs counterpart. We link the degraded accuracy to the vanishing of gradient on the temporal dimension through the analytical and experimental study of the training process of Leaky Integrate-and-Fire (LIF) Neuron-based SNNs. Moreover, we propose the Complementary Leaky Integrate-and-Fire (CLIF) Neuron. CLIF creates extra paths to facilitate the backpropagation in computing temporal gradient while keeping binary output. CLIF is hyperparameter-free and features broad applicability. Extensive experiments on a variety of datasets demonstrate CLIF's clear performance advantage over other neuron models. Furthermore, the CLIF's performance even slightly surpasses superior ANNs with identical network structure and training conditions. The code is available at https://github.com/HuuYuLong/Complementary-LIF.

Figures (11)

• Figure 1: (a) Illustration of the LIF neuron model with forward propagation data flow (b) Illustration of the CLIF neuron model with forward propagation data flow (c) Illustration of the LIF’s gradient error $\frac{\partial\mathcal{L}}{\partial \boldsymbol{u}^{l}[t]}$ flow during BPTT. Each path is represented by an arrow. Lighter color in the arrow indicates more decay of gradient error. (d) Illustration of the CLIF’s gradient error flow during BPTT. Compared to (c), the additional temporal gradient error is highlighted in red.
• Figure 2: The performance of LIF based a 5-layer SNN for CIFAR10 dataset: (a) The accuracy is influenced by time constant ($\tau$) and BPTT timestep ($k$) (detaching all gradients from $k+1$ to T during training). We set the timestep to 6. (b) Accuracy over increasing timestep for both vanilla LIF and our proposed CLIF.
• Figure 3: (a) Loss function vs epochs. Each color presents a case of either LIF, CLIF, or exchanging from LIF to CLIF at a given epoch during training. (b) Comparison of the accuracy of LIF and CLIF at various timestep. Both experiments are evaluated on the CIFAR10 task with Spiking ResNet-18.
• Figure 4: Comparative accuracy of Spiking ResNet-18. Panels (a) CIFAR10 using 8 timestep (b) CIFAR100 using 6 timestep with different neuron.
• Figure 5: Illustration of the LIF neuron based SNN’s gradient error flow during BPTT. In this example k=2: only the backpropagation from the first two timestep is considered (illustrated by the two red dashed arrows), and backpropagation along further timestep is discarded.

Theorems & Definitions (2)

• proof
• proof