Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Training Spiking Neural Networks via Augmented Direct Feedback Alignment

Yongbo Zhang, Katsuma Inoue, Mitsumasa Nakajima, Toshikazu Hashimoto, Yasuo Kuniyoshi, Kohei Nakajima

TL;DR

The proposed aDFA-SNNs scheme can achieve competitive performance without accurate prior knowledge about the utilized system, thus providing a valuable reference for physically training SNNs and demonstrating its superiority and stability over BP and conventional direct feedback alignment.

Abstract

Spiking neural networks (SNNs), the models inspired by the mechanisms of real neurons in the brain, transmit and represent information by employing discrete action potentials or spikes. The sparse, asynchronous properties of information processing make SNNs highly energy efficient, leading to SNNs being promising solutions for implementing neural networks in neuromorphic devices. However, the nondifferentiable nature of SNN neurons makes it a challenge to train them. The current training methods of SNNs that are based on error backpropagation (BP) and precisely designing surrogate gradient are difficult to implement and biologically implausible, hindering the implementation of SNNs on neuromorphic devices. Thus, it is important to train SNNs with a method that is both physically implementatable and biologically plausible. In this paper, we propose using augmented direct feedback alignment (aDFA), a gradient-free approach based on random projection, to train SNNs. This method requires only partial information of the forward process during training, so it is easy to implement and biologically plausible. We systematically demonstrate the feasibility of the proposed aDFA-SNNs scheme, propose its effective working range, and analyze its well-performing settings by employing genetic algorithm. We also analyze the impact of crucial features of SNNs on the scheme, thus demonstrating its superiority and stability over BP and conventional direct feedback alignment. Our scheme can achieve competitive performance without accurate prior knowledge about the utilized system, thus providing a valuable reference for physically training SNNs.

Training Spiking Neural Networks via Augmented Direct Feedback Alignment

TL;DR

The proposed aDFA-SNNs scheme can achieve competitive performance without accurate prior knowledge about the utilized system, thus providing a valuable reference for physically training SNNs and demonstrating its superiority and stability over BP and conventional direct feedback alignment.

Abstract

Spiking neural networks (SNNs), the models inspired by the mechanisms of real neurons in the brain, transmit and represent information by employing discrete action potentials or spikes. The sparse, asynchronous properties of information processing make SNNs highly energy efficient, leading to SNNs being promising solutions for implementing neural networks in neuromorphic devices. However, the nondifferentiable nature of SNN neurons makes it a challenge to train them. The current training methods of SNNs that are based on error backpropagation (BP) and precisely designing surrogate gradient are difficult to implement and biologically implausible, hindering the implementation of SNNs on neuromorphic devices. Thus, it is important to train SNNs with a method that is both physically implementatable and biologically plausible. In this paper, we propose using augmented direct feedback alignment (aDFA), a gradient-free approach based on random projection, to train SNNs. This method requires only partial information of the forward process during training, so it is easy to implement and biologically plausible. We systematically demonstrate the feasibility of the proposed aDFA-SNNs scheme, propose its effective working range, and analyze its well-performing settings by employing genetic algorithm. We also analyze the impact of crucial features of SNNs on the scheme, thus demonstrating its superiority and stability over BP and conventional direct feedback alignment. Our scheme can achieve competitive performance without accurate prior knowledge about the utilized system, thus providing a valuable reference for physically training SNNs.
Paper Structure (15 sections, 17 equations, 8 figures, 2 tables)

This paper contains 15 sections, 17 equations, 8 figures, 2 tables.

Table of Contents

  1. Introduction
  2. aDFA: BP, gradient-free training mechnism
  3. Results
  4. Conclusion
  5. Leaky-integrate-and-fire neuron
  6. Experimental Setup
  7. Initialization of matrices
  8. Accurate differentiable approximation of LIF neurons
  9. Correlation coefficient
  10. Using genetic algorithm to obtain well-performing settings
  11. Impact of the network scale
  12. Impact of the temporal dynamics
  13. Explorations of the performance of backward functions with fixed nonlinear form
  14. Preliminary analysis of parameters of "Opto" and Gaussian based frameworks
  15. The comparison with the performance of existing studies.

Figures (8)

  • Figure 1: The schematics of SNN neuron and dynamics of leaky integrate and fire (LIF) model.(a) SNN neurons transmit discrete signals. (b) Presynaptic spikes are transmitted to postsynaptic neurons, leading to the accumulation of membrane potential, and the postsynaptic spikes are generated when the membrane potential exceeds the firing threshold. After this, the membrane potential is placed to reset the value, and SNN neurons enter a refractory period.
  • Figure 2: Information flow of BP, DFA, and aDFA.(a) BP, transmits the error signal layer by layer, needs to calculate precise $W^{T}$ and derivative $f'$. (b) Orange represents FA, where $W^{T}$ is replaced by fixed, randomly initialized matrices $B$. Red stands for DFA, which injects the global error from the last layer directly to each previous layer through fixed and random matrices $B$. Blue stands for aDFA, a drastic augmentation of DFA, substitutes for $f'$ by arbitrary nonlinear functions $g$.
  • Figure 3: Feasibility of the aDFA-SNNs scheme.(a) The distribution of the correlation coefficient $\eta$ between PRFSs and $f'$ at different orders of scaling factor $\omega$. The $x$ axis represents values of $\eta$, and the $y$ axis denotes the probability density of distribution. (b) The test accuracy on MNIST task as a function of $\eta$ between $f'$ and PRFSs. The whiskers, the line in the middle of the box, and the filled area indicate the maximum and minimum values, the average value, and the distribution density, respectively. The dashed lines indicate the best performances of standard BP and DFA in five trials, which are 97.78% and 96.75%, respectively. (c) Figures of the corresponding shapes of PRFSs in each interval. The blue and orange lines represent selected PRFSs for aDFA and BP, respectively, and the gray line represents $f'$.
  • Figure A.1: The results of the genetic algorithm (GA) optimizing and evolving PRFSs. The left figures denote the fitness score as a function of generation, showing the evolutionary processes of PRFSs. The fitness score is represented by the test accuracy. The box, whisker, and orange line represent the distribution, maximum and minimum values, and median of the population's fitness score, respectively. The right figures represent the shape of PRFSs for randomly initialized and final generation. The red line represents the smoother approximation derivative $f'$, the gray line represents the PRFSs in the population, and the blue line represents the best-performing individual PRFSs. (a) Results on MNIST. (b) Results on F-MNIST.
  • Figure A.2: The results of the impact of network size on the performances of aDFA-SNNs scheme. The size of the network is represented by number of nodes in hidden layer. The box plots show the data distribution of frameworks. Whiskers, orange lines, box bodies, and dots represent the maximum and minimum values, median, data distribution, and outliers, respectively. The line chart illustrates the mean test accuracy of examined frameworks. (a) The results on the MNIST task. (b) The results on the F-MNIST task.
  • ...and 3 more figures