Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spike Accumulation Forwarding for Effective Training of Spiking Neural Networks

Ryuji Saiin, Tomoya Shirakawa, Sota Yoshihara, Yoshihide Sawada, Hiroyuki Kusumoto

TL;DR

The paper introduces Spike Accumulation Forwarding (SAF), a memory- and computation-efficient paradigm for training spiking neural networks by propagating (weighted) spike accumulation and potential accumulation during forward passes. SAF comes in two flavors: SAF-E, which updates parameters at each time step and is shown to be equivalent to OTTT$_{\rm O}$, and SAF-F, which updates only at the final time step and is equivalent to Spike Representation; both are proven to be consistent with established methods. The authors demonstrate that SAF halves forward operations and reduces memory usage while preserving accuracy, and validate this on CIFAR-10/100 with LIF/IF neurons and various connectivity patterns, including feedforward and feedback. The work bridges Spike Representation and OTTT, enabling practical online training of SNNs on GPUs and establishing a path toward efficient inference on standard SNNs, with future work extending to time-dependent data and neuromorphic hardware mapping.

Abstract

In this article, we propose a new paradigm for training spiking neural networks (SNNs), spike accumulation forwarding (SAF). It is known that SNNs are energy-efficient but difficult to train. Consequently, many researchers have proposed various methods to solve this problem, among which online training through time (OTTT) is a method that allows inferring at each time step while suppressing the memory cost. However, to compute efficiently on GPUs, OTTT requires operations with spike trains and weighted summation of spike trains during forwarding. In addition, OTTT has shown a relationship with the Spike Representation, an alternative training method, though theoretical agreement with Spike Representation has yet to be proven. Our proposed method can solve these problems; namely, SAF can halve the number of operations during the forward process, and it can be theoretically proven that SAF is consistent with the Spike Representation and OTTT, respectively. Furthermore, we confirmed the above contents through experiments and showed that it is possible to reduce memory and training time while maintaining accuracy.

Spike Accumulation Forwarding for Effective Training of Spiking Neural Networks

TL;DR

The paper introduces Spike Accumulation Forwarding (SAF), a memory- and computation-efficient paradigm for training spiking neural networks by propagating (weighted) spike accumulation and potential accumulation during forward passes. SAF comes in two flavors: SAF-E, which updates parameters at each time step and is shown to be equivalent to OTTT, and SAF-F, which updates only at the final time step and is equivalent to Spike Representation; both are proven to be consistent with established methods. The authors demonstrate that SAF halves forward operations and reduces memory usage while preserving accuracy, and validate this on CIFAR-10/100 with LIF/IF neurons and various connectivity patterns, including feedforward and feedback. The work bridges Spike Representation and OTTT, enabling practical online training of SNNs on GPUs and establishing a path toward efficient inference on standard SNNs, with future work extending to time-dependent data and neuromorphic hardware mapping.

Abstract

In this article, we propose a new paradigm for training spiking neural networks (SNNs), spike accumulation forwarding (SAF). It is known that SNNs are energy-efficient but difficult to train. Consequently, many researchers have proposed various methods to solve this problem, among which online training through time (OTTT) is a method that allows inferring at each time step while suppressing the memory cost. However, to compute efficiently on GPUs, OTTT requires operations with spike trains and weighted summation of spike trains during forwarding. In addition, OTTT has shown a relationship with the Spike Representation, an alternative training method, though theoretical agreement with Spike Representation has yet to be proven. Our proposed method can solve these problems; namely, SAF can halve the number of operations during the forward process, and it can be theoretically proven that SAF is consistent with the Spike Representation and OTTT, respectively. Furthermore, we confirmed the above contents through experiments and showed that it is possible to reduce memory and training time while maintaining accuracy.
Paper Structure (29 sections, 8 theorems, 75 equations, 12 figures, 6 tables, 1 algorithm)

This paper contains 29 sections, 8 theorems, 75 equations, 12 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Typical Neuron Model
  5. Training methods for SNNs
  6. Spike Accumulation Forwarding
  7. Details of SAF
  8. Equivalence with OTTT$_{\rm O}$ and Spike Representation
  9. Feedforward and Feedback Connection
  10. Experiments
  11. Analysis of SAF-E
  12. Analysis of SAF-F
  13. Limitation and Discussion
  14. Conclusion and Future Work
  15. List of main Formulas
  16. ...and 14 more sections

Key Result

Theorem 1

The backward processes of SAF-E and OTTT$_{\rm O}$ are identical, that is, $\dfrac{\partial L_E[t]}{\partial \bm{W}^l}=\left(\dfrac{\partial L_E[t]}{\partial \bm{W}^l}\right)_{\rm OT}$.

Figures (12)

  • Figure 1: OTTT
  • Figure 2: SAF
  • Figure 4: Accuracy and loss curves of SAF-E and OTTT$_{\rm O}$ on CIFAR-10 ($T=6$).
  • Figure 5: (A) Training Time
  • Figure 6: (B) Memory Consumption
  • ...and 7 more figures

Theorems & Definitions (12)

  • Theorem 1
  • Theorem 2
  • Corollary 3
  • Theorem 4
  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • Corollary 3
  • proof
  • ...and 2 more