Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Three factor delay learning rules for spiking neural networks

Luke Vassallo, Nima Taherinejad

TL;DR

This work tackles the limitation of SNNs by enabling online learning of temporal delays, not just synaptic weights. It introduces three-factor delay learning rules for synaptic and axonal delays in LIF-based networks, using a Gaussian spike-train kernel and a top-down error signal to compute online updates via eligibility traces. Empirical results show that incorporating delays improves accuracy up to 20% over weights-only baselines, with delays achieving close parity to offline backpropagation on SHD when combined with weights, while delivering substantial reductions in model size (6.6x) and inference latency (67%). Overall, the approach supports on-device learning in neuromorphic hardware, offering favorable accuracy/parameter trade-offs and scalable memory requirements through selective delay learning and sparsity.

Abstract

Spiking Neural Networks (SNNs) are dynamical systems that operate on spatiotemporal data, yet their learnable parameters are often limited to synaptic weights, contributing little to temporal pattern recognition. Learnable parameters that delay spike times can improve classification performance in temporal tasks, but existing methods rely on large networks and offline learning, making them unsuitable for real-time operation in resource-constrained environments. In this paper, we introduce synaptic and axonal delays to leaky integrate and fire (LIF)-based feedforward and recurrent SNNs, and propose three-factor learning rules to simultaneously learn delay parameters online. We employ a smooth Gaussian surrogate to approximate spike derivatives exclusively for the eligibility trace calculation, and together with a top-down error signal determine parameter updates. Our experiments show that incorporating delays improves accuracy by up to 20% over a weights-only baseline, and for networks with similar parameter counts, jointly learning weights and delays yields up to 14% higher accuracy. On the SHD speech recognition dataset, our method achieves similar accuracy to offline backpropagation-based approaches. Compared to state-of-the-art methods, it reduces model size by 6.6x and inference latency by 67%, with only a 2.4% drop in classification accuracy. Our findings benefit the design of power and area-constrained neuromorphic processors by enabling on-device learning and lowering memory requirements.

Three factor delay learning rules for spiking neural networks

TL;DR

This work tackles the limitation of SNNs by enabling online learning of temporal delays, not just synaptic weights. It introduces three-factor delay learning rules for synaptic and axonal delays in LIF-based networks, using a Gaussian spike-train kernel and a top-down error signal to compute online updates via eligibility traces. Empirical results show that incorporating delays improves accuracy up to 20% over weights-only baselines, with delays achieving close parity to offline backpropagation on SHD when combined with weights, while delivering substantial reductions in model size (6.6x) and inference latency (67%). Overall, the approach supports on-device learning in neuromorphic hardware, offering favorable accuracy/parameter trade-offs and scalable memory requirements through selective delay learning and sparsity.

Abstract

Spiking Neural Networks (SNNs) are dynamical systems that operate on spatiotemporal data, yet their learnable parameters are often limited to synaptic weights, contributing little to temporal pattern recognition. Learnable parameters that delay spike times can improve classification performance in temporal tasks, but existing methods rely on large networks and offline learning, making them unsuitable for real-time operation in resource-constrained environments. In this paper, we introduce synaptic and axonal delays to leaky integrate and fire (LIF)-based feedforward and recurrent SNNs, and propose three-factor learning rules to simultaneously learn delay parameters online. We employ a smooth Gaussian surrogate to approximate spike derivatives exclusively for the eligibility trace calculation, and together with a top-down error signal determine parameter updates. Our experiments show that incorporating delays improves accuracy by up to 20% over a weights-only baseline, and for networks with similar parameter counts, jointly learning weights and delays yields up to 14% higher accuracy. On the SHD speech recognition dataset, our method achieves similar accuracy to offline backpropagation-based approaches. Compared to state-of-the-art methods, it reduces model size by 6.6x and inference latency by 67%, with only a 2.4% drop in classification accuracy. Our findings benefit the design of power and area-constrained neuromorphic processors by enabling on-device learning and lowering memory requirements.
Paper Structure (18 sections, 12 equations, 3 figures, 1 table)

This paper contains 18 sections, 12 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Delay learning
  4. Online learning
  5. Proposed Methods
  6. Neuron dynamics
  7. Surrogate gradient
  8. Neural network architecture and loss function
  9. Online learnable delays
  10. Spike train kernel
  11. Experiments
  12. Dataset
  13. Experimental Setup
  14. Results
  15. Comparison and Discussion
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Illustration of coinciding pre-synaptic spikes leading to post-synaptic. Two distinct pre-synaptic neurons emit spikes at $t=t_1$ and $t=t_2$, respectively. Introducing a delay to the first spike shifts its effect to $t=t_{12}$, where it coincides with the second spike and thereby triggering downstream effects.
  • Figure 2: Comparison against offline methods on and (gray background). Marker shape distinguishes model type, while color determines the type and presence of delays. Markers overlaid with an X correspond to the proposed online method
  • Figure 3: Control experiments illustrating the efficacy of delay-learning. (a, b) Sweep model sparsity for axonal (AD) and synaptic delays (SD), with fixed (W+F*) or learnable (W+L*) delay parameters. (c) Compares learning delays with fixed weights (F-W*) using the proposed method and . (d) Compares with delays in the input, recurrent, and all connections.