Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scaling Up Resonate-and-Fire Networks for Fast Deep Learning

Thomas E. Huber, Jules Lecomte, Borislav Polovnikov, Axel von Arnim

TL;DR

S5-RF is introduced, a new SSM layer comprised of RF neurons based on the S5 model, that features a generic initialization scheme and fast training within a deep architecture that achieves similar performance with much fewer spiking operations.

Abstract

Spiking neural networks (SNNs) present a promising computing paradigm for neuromorphic processing of event-based sensor data. The resonate-and-fire (RF) neuron, in particular, appeals through its biological plausibility, complex dynamics, yet computational simplicity. Despite theoretically predicted benefits, challenges in parameter initialization and efficient learning inhibited the implementation of RF networks, constraining their use to a single layer. In this paper, we address these shortcomings by deriving the RF neuron as a structured state space model (SSM) from the HiPPO framework. We introduce S5-RF, a new SSM layer comprised of RF neurons based on the S5 model, that features a generic initialization scheme and fast training within a deep architecture. S5-RF scales for the first time a RF network to a deep SNN with up to four layers and achieves with 78.8% a new state-of-the-art result for recurrent SNNs on the Spiking Speech Commands dataset in under three hours of training time. Moreover, compared to the reference SNNs that solve our benchmarking tasks, it achieves similar performance with much fewer spiking operations. Our code is publicly available at https://github.com/ThomasEHuber/s5-rf.

Scaling Up Resonate-and-Fire Networks for Fast Deep Learning

TL;DR

S5-RF is introduced, a new SSM layer comprised of RF neurons based on the S5 model, that features a generic initialization scheme and fast training within a deep architecture that achieves similar performance with much fewer spiking operations.

Abstract

Spiking neural networks (SNNs) present a promising computing paradigm for neuromorphic processing of event-based sensor data. The resonate-and-fire (RF) neuron, in particular, appeals through its biological plausibility, complex dynamics, yet computational simplicity. Despite theoretically predicted benefits, challenges in parameter initialization and efficient learning inhibited the implementation of RF networks, constraining their use to a single layer. In this paper, we address these shortcomings by deriving the RF neuron as a structured state space model (SSM) from the HiPPO framework. We introduce S5-RF, a new SSM layer comprised of RF neurons based on the S5 model, that features a generic initialization scheme and fast training within a deep architecture. S5-RF scales for the first time a RF network to a deep SNN with up to four layers and achieves with 78.8% a new state-of-the-art result for recurrent SNNs on the Spiking Speech Commands dataset in under three hours of training time. Moreover, compared to the reference SNNs that solve our benchmarking tasks, it achieves similar performance with much fewer spiking operations. Our code is publicly available at https://github.com/ThomasEHuber/s5-rf.

Paper Structure

This paper contains 17 sections, 15 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Resonate-and-Fire Neurons
  4. Structured State Space Models
  5. Definition
  6. Discretization
  7. HiPPO-Initialization
  8. Our Approach: S5-RF
  9. From SSM to Deep RF
  10. Dirac-SSM Discretization
  11. Spiking without Losing Speed
  12. Experiments
  13. Sequential MNIST
  14. Audio Processing
  15. Results
  16. ...and 2 more sections

Figures (3)

  • Figure 1: (a) S5-RF layer architecture based upon the S5 architecture smith-S5. (b) Imaginary values of the components of $\Lambda_{N}$ in the case of $H=32$ dimensions. We note that since the matrix $A_{N}$ is real, its eigenvalues necessarily come in complex-conjugated pairs.
  • Figure 2: Spiking activity of in- and output of a S5-RF layer trained on SHD. (a) Sample input of the word "nine" with locally dense spiking activity across neurons. (b) Output of a layer with some neurons bursting and others remaining silent.
  • Figure 3: Ablation results in a 512x2 S5-RF network. (a) Test accuracy for the HiPPO initialization with fixed (bottom line, blue) and learned (top line, red) $\eta$, respectively. The shaded area around the curves represents the standard deviation obtained from five different random seeds. (b) Test accuracy for random initialization.