Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SpikePipe: Accelerated Training of Spiking Neural Networks via Inter-Layer Pipelining and Multiprocessor Scheduling

Sai Sanjeet, Bibhu Datta Sahoo, Keshab K. Parhi

TL;DR

Inter-layer pipelining to accelerate training in SNNs using systolic array-based processors and multiprocessor scheduling is proposed, and the results show that the proposed method achieves an average speedup of 1.7 compared to standard pipelining algorithms.

Abstract

Spiking Neural Networks (SNNs) have gained popularity due to their high energy efficiency. Prior works have proposed various methods for training SNNs, including backpropagation-based methods. Training SNNs is computationally expensive compared to their conventional counterparts and would benefit from multiprocessor hardware acceleration. This is the first paper to propose inter-layer pipelining to accelerate training in SNNs using systolic array-based processors and multiprocessor scheduling. The impact of training using delayed gradients is observed using three networks training on different datasets, showing no degradation for small networks and < 10% degradation for large networks. The mapping of various training tasks of the SNN onto systolic arrays is formulated, and the proposed scheduling method is evaluated on the three networks. The results are compared against standard pipelining algorithms. The results show that the proposed method achieves an average speedup of 1.6X compared to standard pipelining algorithms, with an upwards of 2X improvement in some cases. The incurred communication overhead due to the proposed method is less than 0.5% of the total required communication of training.

SpikePipe: Accelerated Training of Spiking Neural Networks via Inter-Layer Pipelining and Multiprocessor Scheduling

TL;DR

Inter-layer pipelining to accelerate training in SNNs using systolic array-based processors and multiprocessor scheduling is proposed, and the results show that the proposed method achieves an average speedup of 1.7 compared to standard pipelining algorithms.

Abstract

Spiking Neural Networks (SNNs) have gained popularity due to their high energy efficiency. Prior works have proposed various methods for training SNNs, including backpropagation-based methods. Training SNNs is computationally expensive compared to their conventional counterparts and would benefit from multiprocessor hardware acceleration. This is the first paper to propose inter-layer pipelining to accelerate training in SNNs using systolic array-based processors and multiprocessor scheduling. The impact of training using delayed gradients is observed using three networks training on different datasets, showing no degradation for small networks and < 10% degradation for large networks. The mapping of various training tasks of the SNN onto systolic arrays is formulated, and the proposed scheduling method is evaluated on the three networks. The results are compared against standard pipelining algorithms. The results show that the proposed method achieves an average speedup of 1.6X compared to standard pipelining algorithms, with an upwards of 2X improvement in some cases. The incurred communication overhead due to the proposed method is less than 0.5% of the total required communication of training.
Paper Structure (24 sections, 28 equations, 18 figures, 7 tables, 1 algorithm)

This paper contains 24 sections, 28 equations, 18 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Modeling Neurons for SNNs
  3. Training SNNs
  4. Loss Function
  5. Backpropagation
  6. $i_{in}[n]$
  7. $v_{m}[n]$
  8. $v_{sp}[n]$
  9. Dataflow Graph of Training
  10. Delayed Gradients
  11. Example Networks
  12. Accelerators for Training SNNs
  13. Modified systolic array for SNNs
  14. Estimating clock cycles for computation
  15. Convolutional layer
  16. ...and 9 more sections

Figures (18)

  • Figure 1: Digital first-order section for implementing the LIF neuron model.
  • Figure 2: Modified LIF structure incorporating the reset operation. The delay element is reset when the membrane potential crosses the threshold voltage.
  • Figure 3: (a) Sample input current for 20 timesteps and (b) the corresponding membrane potential and spike train. The values of $c$ and $\lambda$ are 4 and 0.25, respectively, and the threshold voltage is 0.5.
  • Figure 4: A sample spiking neural network with two layers.
  • Figure 5: Dataflow graph of a neuron in the hidden layer for $T$ timesteps. Spiking of the neuron in the previous timestep gates the input current and membrane potential of the current timestep, and is shown by the colored line.
  • ...and 13 more figures