Toward Large-scale Spiking Neural Networks: A Comprehensive Survey and Future Directions

TL;DR

This survey examines energy-efficient pathways for large-scale AI via Deep Spiking Neural Networks, focusing on two core learning rules—ANN-to-SNN conversion and surrogate-gradient-based direct training—and two architectural families—DCNNs and Spiking Transformers. It reviews key techniques that bridge ANN and SNN performance, including normalization, thresholding, and surrogate gradient methods, and surveys the rise of Spiking Transformers such as Spikformer and QKFormer, which narrow the accuracy gap with ANN baselines at low latency. Benchmarking on ImageNet shows substantial progress, with 4-time-step SNNs approaching ANN performance and directionality toward even higher accuracy with efficient attention and architecture search. The paper also outlines future directions toward large SNNs,multi-modal capabilities, and new learning paradigms that could sustain energy-efficient, scalable neuromorphic AI in a variety of tasks.

Abstract

Deep learning has revolutionized artificial intelligence (AI), achieving remarkable progress in fields such as computer vision, speech recognition, and natural language processing. Moreover, the recent success of large language models (LLMs) has fueled a surge in research on large-scale neural networks. However, the escalating demand for computing resources and energy consumption has prompted the search for energy-efficient alternatives. Inspired by the human brain, spiking neural networks (SNNs) promise energy-efficient computation with event-driven spikes. To provide future directions toward building energy-efficient large SNN models, we present a survey of existing methods for developing deep spiking neural networks, with a focus on emerging Spiking Transformers. Our main contributions are as follows: (1) an overview of learning methods for deep spiking neural networks, categorized by ANN-to-SNN conversion and direct training with surrogate gradients; (2) an overview of network architectures for deep spiking neural networks, categorized by deep convolutional neural networks (DCNNs) and Transformer architecture; and (3) a comprehensive comparison of state-of-the-art deep SNNs with a focus on emerging Spiking Transformers. We then further discuss and outline future directions toward large-scale SNNs.

Figures (5)

• Figure 1: ANN-to-SNN conversion, a mapping between real-valued activation neurons and spiking neurons.
• Figure 2: Surrogate gradient function (linear) for backpropagation.
• Figure 3: Different residual connections in SNNs.
• Figure 4: An example of spiking self attention.
• Figure 5: ImageNet classification results for SOTA Spiking Transformers.