Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

STEP: A Unified Spiking Transformer Evaluation Platform for Fair and Reproducible Benchmarking

Sicheng Shen, Dongcheng Zhao, Linghao Feng, Zeyang Yue, Jindong Li, Tenglong Li, Guobin Shen, Yi Zeng

TL;DR

STEP addresses the lack of fair, reproducible benchmarking for Spiking Transformers by introducing a modular, backend-agnostic platform that supports classification, segmentation, and detection across static, event-based, and sequential data. It enables unified reproduction of representative models and systematic ablations of neurons, encodings, and attention, alongside a unified energy-efficiency model that accounts for memory and bitwidth. Key findings show current Spiking Transformers rely heavily on convolutional frontends with limited temporal modeling, while quantized ANNs can be competitive in energy when memory costs are considered. STEP provides a foundation for developing spike-native, temporally principled architectures and promotes principled, scalable evaluation in neuromorphic vision.

Abstract

Spiking Transformers have recently emerged as promising architectures for combining the efficiency of spiking neural networks with the representational power of self-attention. However, the lack of standardized implementations, evaluation pipelines, and consistent design choices has hindered fair comparison and principled analysis. In this paper, we introduce STEP a unified benchmark framework for Spiking Transformers that supports a wide range of tasks, including classification, segmentation, and detection across static, event-based, and sequential datasets. STEP provides modular support for diverse components such as spiking neurons, input encodings, surrogate gradients, and multiple backends (e.g., SpikingJelly, BrainCog). Using STEP, we reproduce and evaluate several representative models, and conduct systematic ablation studies on attention design, neuron types, encoding schemes, and temporal modeling capabilities. We also propose a unified analytical model for energy estimation, accounting for spike sparsity, bitwidth, and memory access, and show that quantized ANNs may offer comparable or better energy efficiency. Our results suggest that current Spiking Transformers rely heavily on convolutional frontends and lack strong temporal modeling, underscoring the need for spike-native architectural innovations. The full code is available at: https://github.com/Fancyssc/STEP

STEP: A Unified Spiking Transformer Evaluation Platform for Fair and Reproducible Benchmarking

TL;DR

STEP addresses the lack of fair, reproducible benchmarking for Spiking Transformers by introducing a modular, backend-agnostic platform that supports classification, segmentation, and detection across static, event-based, and sequential data. It enables unified reproduction of representative models and systematic ablations of neurons, encodings, and attention, alongside a unified energy-efficiency model that accounts for memory and bitwidth. Key findings show current Spiking Transformers rely heavily on convolutional frontends with limited temporal modeling, while quantized ANNs can be competitive in energy when memory costs are considered. STEP provides a foundation for developing spike-native, temporally principled architectures and promotes principled, scalable evaluation in neuromorphic vision.

Abstract

Spiking Transformers have recently emerged as promising architectures for combining the efficiency of spiking neural networks with the representational power of self-attention. However, the lack of standardized implementations, evaluation pipelines, and consistent design choices has hindered fair comparison and principled analysis. In this paper, we introduce STEP a unified benchmark framework for Spiking Transformers that supports a wide range of tasks, including classification, segmentation, and detection across static, event-based, and sequential datasets. STEP provides modular support for diverse components such as spiking neurons, input encodings, surrogate gradients, and multiple backends (e.g., SpikingJelly, BrainCog). Using STEP, we reproduce and evaluate several representative models, and conduct systematic ablation studies on attention design, neuron types, encoding schemes, and temporal modeling capabilities. We also propose a unified analytical model for energy estimation, accounting for spike sparsity, bitwidth, and memory access, and show that quantized ANNs may offer comparable or better energy efficiency. Our results suggest that current Spiking Transformers rely heavily on convolutional frontends and lack strong temporal modeling, underscoring the need for spike-native architectural innovations. The full code is available at: https://github.com/Fancyssc/STEP
Paper Structure (51 sections, 11 equations, 8 figures, 14 tables)

This paper contains 51 sections, 11 equations, 8 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. SNN Input Encoding
  4. Spiking Neurons
  5. Spiking Self-Attention
  6. Other Modules
  7. Recent Advancements
  8. Spiking Transformer Benchmark
  9. Flexible and Modular Architecture
  10. Broad Dataset Compatibility
  11. Multi-task Adaptation
  12. Backend-Agnostic Integration
  13. Experiment
  14. Reproduction
  15. Experiments on More Complex Tasks
  16. ...and 36 more sections

Figures (8)

  • Figure 1: Unified Spiking Transformer Framework with Flexible Encoding, Attention Modules, and Application-specific Heads
  • Figure 2: System Architecture of STEP as a Unified Benchmark for Spiking Transformer Development and Evaluation
  • Figure 3: Segmentation predictions on ADE20K for three Spiking Transformer variants.
  • Figure 4: Result of SDTv2 on COCO datasets.
  • Figure 5: Visualization of different encoding methods.
  • ...and 3 more figures