Spiking Deep Networks with LIF Neurons

TL;DR

This work addresses training deep networks with biologically realistic LIF spiking neurons while maintaining competitive accuracy on standard image datasets. It introduces a softening of the LIF nonlinearity to bound backpropagated derivatives and uses noise during training to mimic spike-induced variability, enabling effective backpropagation. The static CNN is converted to a spiking network with minimal changes, achieving state-of-the-art results for spiking models on CIFAR-10 and strong MNIST performance, and it discusses implications for neuromorphic hardware. Overall, the approach demonstrates that biologically plausible spiking neurons can be integrated into deep networks without sacrificing performance, with potential benefits for energy-efficient image classification.

Abstract

We train spiking deep networks using leaky integrate-and-fire (LIF) neurons, and achieve state-of-the-art results for spiking networks on the CIFAR-10 and MNIST datasets. This demonstrates that biologically-plausible spiking LIF neurons can be integrated into deep networks can perform as well as other spiking models (e.g. integrate-and-fire). We achieved this result by softening the LIF response function, such that its derivative remains bounded, and by training the network with noise to provide robustness against the variability introduced by spikes. Our method is general and could be applied to other neuron types, including those used on modern neuromorphic hardware. Our work brings more biological realism into modern image classification models, with the hope that these models can inform how the brain performs this difficult task. It also provides new methods for training deep networks to run on neuromorphic hardware, with the aim of fast, power-efficient image classification for robotics applications.

Figures (2)

• Figure 1: Comparison of LIF and soft LIF response functions. The left panel shows the response functions themselves. The LIF function has a hard threshold at $j = V_{th} = 1$; the soft LIF function smooths this threshold. The right panel shows the derivatives of the response functions. The hard LIF function has a discontinuous and unbounded derivative at $j = 1$; the soft LIF function has a continuous bounded derivative, making it amenable to use in backpropagation.
• Figure 2: Variability in filtered spike trains versus input current for the LIF neuron ($\tau_{RC} = 0.02, \tau_{ref} = 0.004$). The solid line shows the mean of the filtered spike train (which matches the analytical rate of Equation \ref{['eqn:lifss']}), the 'x'-points show the median, the solid error bars show the 25th and 75th percentiles, and the dotted error bars show the minimum and maximum. The spike train was filtered with an $\alpha$-filter (Equation \ref{['eqn:alpha']}) with $\tau_s = 0.003$ s. (Note that this is different than the $\tau_s = 0.005$ used in simulation, to better display the variation.)