Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips

TL;DR

The paper introduces Meta-SpikeFormer, a meta Transformer-based SNN architecture tailored for spike-driven, energy-efficient neuromorphic chips. By combining Conv-based SNN blocks with Transformer-based SNN stages and a novel Spike-Driven Self-Attention (SDSA) mechanism, it achieves state-of-the-art results on ImageNet-1K (80.0% top-1 with 55M parameters) and demonstrates versatility across classification, detection, and segmentation. The work highlights how a meta-architecture, SDSA variants, and careful shortcut design can enable high performance with sparse, spike-driven computation, offering guidance for future neuromorphic chip design. Extensive experiments and ablations illustrate the trade-offs between architecture choices, energy consumption, and accuracy, and position Meta-SpikeFormer as a strong universal backbone for SNN-based vision tasks. This work also frames a pathway for algorithm-hardware co-design in next-generation Transformer-based neuromorphic chips.

Abstract

Neuromorphic computing, which exploits Spiking Neural Networks (SNNs) on neuromorphic chips, is a promising energy-efficient alternative to traditional AI. CNN-based SNNs are the current mainstream of neuromorphic computing. By contrast, no neuromorphic chips are designed especially for Transformer-based SNNs, which have just emerged, and their performance is only on par with CNN-based SNNs, offering no distinct advantage. In this work, we propose a general Transformer-based SNN architecture, termed as ``Meta-SpikeFormer", whose goals are: 1) Lower-power, supports the spike-driven paradigm that there is only sparse addition in the network; 2) Versatility, handles various vision tasks; 3) High-performance, shows overwhelming performance advantages over CNN-based SNNs; 4) Meta-architecture, provides inspiration for future next-generation Transformer-based neuromorphic chip designs. Specifically, we extend the Spike-driven Transformer in \citet{yao2023spike} into a meta architecture, and explore the impact of structure, spike-driven self-attention, and skip connection on its performance. On ImageNet-1K, Meta-SpikeFormer achieves 80.0\% top-1 accuracy (55M), surpassing the current state-of-the-art (SOTA) SNN baselines (66M) by 3.7\%. This is the first direct training SNN backbone that can simultaneously supports classification, detection, and segmentation, obtaining SOTA results in SNNs. Finally, we discuss the inspiration of the meta SNN architecture for neuromorphic chip design. Source code and models are available at \url{https://github.com/BICLab/Spike-Driven-Transformer-V2}.

Figures (4)

• Figure 1: Meta Transformer Block.
• Figure 2: The overview of Meta-SpikeFormer. At the macro level, we refer to the general vision Transformer architecture in yu2022metaformeryu2022metaformer_2 and align Spike-driven Transformer yao2023spike with it. The main macro-level alteration is that we enlarge the spike coding layer from four Conv SNN layers to four Conv-based SNN blocks. At the micro level, we use the meta Transformer block in Fig. \ref{['fig_meta_block']} as the basis to upgrade to Conv-based and Transformer-based SNN blocks (see Table \ref{['table_ablation_study']}), such as Channel Conv, SDSA operations, etc., to bring them more in line with SNN features.
• Figure 3: Spike-Driven Self-Attention (SDSA) modules with different computational complexity. SDSA-1 is the operator used in yao2023spike. SDSA-2/3/4 is the newly designed operator in this paper. We exploit SDSA-3 by default. All SDSAs only have addition, no softmax and scale.
• Figure 4: Existing shortcut in SNNs.