Parallel Spiking Neurons with High Efficiency and Ability to Learn Long-term Dependencies

TL;DR

The paper tackles the serial bottleneck of traditional spiking neurons caused by charge-fire-reset dynamics that impede long-term dependencies. It introduces Parallel Spiking Neuron (PSN) by removing the reset, yielding a fully parallelizable formulation via $\mathbf{H}=\mathbf{W}\mathbf{X}$ and $\mathbf{S}=\Theta(\mathbf{H}-\mathbf{B})$, and extends it with $k$-order Masked PSN and Sliding PSN to accommodate variable-length sequences. Experiments demonstrate dramatic simulation-speed improvements and competitive or superior accuracy on sequential CIFAR10/100, static CIFAR10/ImageNet, and neuromorphic CIFAR10-DVS, while keeping memory footprint small. The work provides a foundational parallelization strategy for SNNs and practical variants for different latency/throughput needs, with code available.

Abstract

Vanilla spiking neurons in Spiking Neural Networks (SNNs) use charge-fire-reset neuronal dynamics, which can only be simulated serially and can hardly learn long-time dependencies. We find that when removing reset, the neuronal dynamics can be reformulated in a non-iterative form and parallelized. By rewriting neuronal dynamics without reset to a general formulation, we propose the Parallel Spiking Neuron (PSN), which generates hidden states that are independent of their predecessors, resulting in parallelizable neuronal dynamics and extremely high simulation speed. The weights of inputs in the PSN are fully connected, which maximizes the utilization of temporal information. To avoid the use of future inputs for step-by-step inference, the weights of the PSN can be masked, resulting in the masked PSN. By sharing weights across time-steps based on the masked PSN, the sliding PSN is proposed to handle sequences of varying lengths. We evaluate the PSN family on simulation speed and temporal/static data classification, and the results show the overwhelming advantage of the PSN family in efficiency and accuracy. To the best of our knowledge, this is the first study about parallelizing spiking neurons and can be a cornerstone for the spiking deep learning research. Our codes are available at \url{https://github.com/fangwei123456/Parallel-Spiking-Neuron}.

Figures (5)

• Figure 1: The computational graphs of the vanilla spiking neuron and the parallel spiking neuron. Figure (a) is cited from fang2021incorporating. The dotted lines in Figure (b) are weights that can be masked for step-by-step computation. $X[t], S[t]$ are the input current and the output spike, and $H[t], V[t]$ are hidden states at time-step $t$.
• Figure 2: Comparison of the PSN family on the parameter number $n_{param}$ and the generation of hidden states.
• Figure 3: The executing time ratio of the LIF neuron and the PSN for one forward in inference and one forward and backward in training.
• Figure 4: The accuracy curves and firing rates of different neurons for classifying sequential CIFAR.
• Figure 5: The accuracy-order curve on the sequential CIFAR100.