Spike-based computation using classical recurrent neural networks

TL;DR

This work tackles the challenge of training energy-efficient spiking neural networks by introducing the Spiking Recurrent Cell (SRC), a differentiable, event-based neuron derived from GRU that enables vanilla backpropagation. SRC combines spike-generation via bistable cellular memory with input integration, allowing deep SRC-based networks to be trained more reliably than traditional LIF-based SNNs. Experiments on MNIST, Fashion-MNIST, and Neuromorphic-MNIST show SRCs achieving competitive performance in shallow networks and enabling training of deeper architectures, with added robustness to input noise. The approach has implications for neuromorphic hardware and scalable, deep spike-based computation, and points toward future extensions such as convolutional SRCs and hardware-friendly implementations.

Abstract

Spiking neural networks are a type of artificial neural networks in which communication between neurons is only made of events, also called spikes. This property allows neural networks to make asynchronous and sparse computations and therefore drastically decrease energy consumption when run on specialised hardware. However, training such networks is known to be difficult, mainly due to the non-differentiability of the spike activation, which prevents the use of classical backpropagation. This is because state-of-the-art spiking neural networks are usually derived from biologically-inspired neuron models, to which are applied machine learning methods for training. Nowadays, research about spiking neural networks focuses on the design of training algorithms whose goal is to obtain networks that compete with their non-spiking version on specific tasks. In this paper, we attempt the symmetrical approach: we modify the dynamics of a well-known, easily trainable type of recurrent neural network to make it event-based. This new RNN cell, called the Spiking Recurrent Cell, therefore communicates using events, i.e. spikes, while being completely differentiable. Vanilla backpropagation can thus be used to train any network made of such RNN cell. We show that this new network can achieve performance comparable to other types of spiking networks in the MNIST benchmark and its variants, the Fashion-MNIST and the Neuromorphic-MNIST. Moreover, we show that this new cell makes the training of deep spiking networks achievable.

Figures (8)

• Figure 1: Simulation of a SRC neuron for some inputs sequence $x$ and different biases $b_h$.
• Figure 2: Evolution of the training cross-entropy and the validating accuracy on the MNIST dataset for two sets of networks composed of 5 SRC layers. For one set, the ReLU has been skipped during the backpropagation.
• Figure 3: Evolution of the training and validating accuracies of SRC networks with 2, 3 and 5 layers on the Fashion MNIST with latency-based coding.
• Figure 4: Mean spiking activity on the test set of the MNIST dataset with rate-based coding (left) and latency-based coding (right).
• Figure 5: (left) Evolution of the cross-entropy (top) and the accuracy (bottom) on the validation set during the training of several networks with 5, 10 and 15 layers. (top, right) Evolution of the epoch duration in seconds of these networks. (bottom, right) Cross-entropies and accuracies obtained on the MNIST test set by the models with respect to their number of hidden layers.