Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Direct Training High-Performance Deep Spiking Neural Networks: A Review of Theories and Methods

Chenlin Zhou, Han Zhang, Liutao Yu, Yumin Ye, Zhaokun Zhou, Liwei Huang, Zhengyu Ma, Xiaopeng Fan, Huihui Zhou, Yonghong Tian

TL;DR

The review addresses the challenge of training deep spiking neural networks (SNNs) directly for high performance, outlining the shift from ANN-to-SNN conversion to surrogate-gradient based training and highlighting transformer-based SNNs that reach or exceed many ANN benchmarks. It consolidates theory, neuron models (including trainable and parallel variants), and training mechanisms (surrogate gradients, loss functions, and normalization), while surveying architectural innovations (transformers and residual blocks) and the software/hardware ecosystem enabling practical deployment. The article further canvasses a wide range of applications across computer vision, reinforcement learning, and neuroscience-inspired domains, noting both the energy-efficiency and temporal-processing advantages of SNNs, as well as current limitations in training cost, energy accounting, and interoperability. By identifying future directions—improved biological realism, astrocyte-inspired computation, advanced time-dependent encoding, scalable training, and standardized interfaces—it articulates a roadmap for advancing deep SNNs toward real-world impact on neuromorphic hardware and energy-efficient AI.

Abstract

Spiking neural networks (SNNs) offer a promising energy-efficient alternative to artificial neural networks (ANNs), in virtue of their high biological plausibility, rich spatial-temporal dynamics, and event-driven computation. The direct training algorithms based on the surrogate gradient method provide sufficient flexibility to design novel SNN architectures and explore the spatial-temporal dynamics of SNNs. According to previous studies, the performance of models is highly dependent on their sizes. Recently, direct training deep SNNs have achieved great progress on both neuromorphic datasets and large-scale static datasets. Notably, transformer-based SNNs show comparable performance with their ANN counterparts. In this paper, we provide a new perspective to summarize the theories and methods for training deep SNNs with high performance in a systematic and comprehensive way, including theory fundamentals, spiking neuron models, advanced SNN models and residual architectures, software frameworks and neuromorphic hardware, applications, and future trends. The reviewed papers are collected at https://github.com/zhouchenlin2096/Awesome-Spiking-Neural-Networks

Direct Training High-Performance Deep Spiking Neural Networks: A Review of Theories and Methods

TL;DR

The review addresses the challenge of training deep spiking neural networks (SNNs) directly for high performance, outlining the shift from ANN-to-SNN conversion to surrogate-gradient based training and highlighting transformer-based SNNs that reach or exceed many ANN benchmarks. It consolidates theory, neuron models (including trainable and parallel variants), and training mechanisms (surrogate gradients, loss functions, and normalization), while surveying architectural innovations (transformers and residual blocks) and the software/hardware ecosystem enabling practical deployment. The article further canvasses a wide range of applications across computer vision, reinforcement learning, and neuroscience-inspired domains, noting both the energy-efficiency and temporal-processing advantages of SNNs, as well as current limitations in training cost, energy accounting, and interoperability. By identifying future directions—improved biological realism, astrocyte-inspired computation, advanced time-dependent encoding, scalable training, and standardized interfaces—it articulates a roadmap for advancing deep SNNs toward real-world impact on neuromorphic hardware and energy-efficient AI.

Abstract

Spiking neural networks (SNNs) offer a promising energy-efficient alternative to artificial neural networks (ANNs), in virtue of their high biological plausibility, rich spatial-temporal dynamics, and event-driven computation. The direct training algorithms based on the surrogate gradient method provide sufficient flexibility to design novel SNN architectures and explore the spatial-temporal dynamics of SNNs. According to previous studies, the performance of models is highly dependent on their sizes. Recently, direct training deep SNNs have achieved great progress on both neuromorphic datasets and large-scale static datasets. Notably, transformer-based SNNs show comparable performance with their ANN counterparts. In this paper, we provide a new perspective to summarize the theories and methods for training deep SNNs with high performance in a systematic and comprehensive way, including theory fundamentals, spiking neuron models, advanced SNN models and residual architectures, software frameworks and neuromorphic hardware, applications, and future trends. The reviewed papers are collected at https://github.com/zhouchenlin2096/Awesome-Spiking-Neural-Networks
Paper Structure (29 sections, 19 equations, 4 figures, 4 tables)

This paper contains 29 sections, 19 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Spiking Neuron Models
  3. Spiking Neurons with Trainable Parameters
  4. Parallel Spiking Neurons
  5. Fundamentals of Spiking Neural Networks
  6. Information Coding
  7. Network Training
  8. Surrogate Gradient
  9. Loss Function and Backpropagation
  10. Batch Normalization
  11. SNN Architecture developments
  12. Transformer-based Spiking Neural Networks
  13. Residual Architectures in Spiking Neural Networks
  14. Others
  15. Software Frameworks and Neuromorphic Hardware for Spiking Neural Networks
  16. ...and 14 more sections

Figures (4)

  • Figure 1: (a) The scheme of a spiking neuron, of which the input and output are both binary spikes. (b) The sigmoid function approximates the Heaviside activation function of a spiking neuron, and its derivative can be utilized to calculate gradients during backpropagation.
  • Figure 2: The backward of BPTT and OTTT.
  • Figure 3: The overview of Spiking Transformer (Spikformer).
  • Figure 4: The overview of residual learning architectures.