Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Neural Dynamics Self-Attention for Spiking Transformers

Dehao Zhang, Fukai Guo, Shuai Wang, Jingya Wang, Jieyuan Zhang, Yimeng Shan, Malu Zhang, Yang Yang, Haizhou Li

Abstract

Integrating Spiking Neural Networks (SNNs) with Transformer architectures offers a promising pathway to balance energy efficiency and performance, particularly for edge vision applications. However, existing Spiking Transformers face two critical challenges: (i) a substantial performance gap compared to their Artificial Neural Networks (ANNs) counterparts and (ii) high memory overhead during inference. Through theoretical analysis, we attribute both limitations to the Spiking Self-Attention (SSA) mechanism: the lack of locality bias and the need to store large attention matrices. Inspired by the localized receptive fields (LRF) and membrane-potential dynamics of biological visual neurons, we propose LRF-Dyn, which uses spiking neurons with localized receptive fields to compute attention while reducing memory requirements. Specifically, we introduce a LRF method into SSA to assign higher weights to neighboring regions, strengthening local modeling and improving performance. Building on this, we approximate the resulting attention computation via charge-fire-reset dynamics, eliminating explicit attention-matrix storage and reducing inference-time memory. Extensive experiments on visual tasks confirm that our method reduces memory overhead while delivering significant performance improvements. These results establish it as a key unit for achieving energy-efficient Spiking Transformers.

Neural Dynamics Self-Attention for Spiking Transformers

Abstract

Integrating Spiking Neural Networks (SNNs) with Transformer architectures offers a promising pathway to balance energy efficiency and performance, particularly for edge vision applications. However, existing Spiking Transformers face two critical challenges: (i) a substantial performance gap compared to their Artificial Neural Networks (ANNs) counterparts and (ii) high memory overhead during inference. Through theoretical analysis, we attribute both limitations to the Spiking Self-Attention (SSA) mechanism: the lack of locality bias and the need to store large attention matrices. Inspired by the localized receptive fields (LRF) and membrane-potential dynamics of biological visual neurons, we propose LRF-Dyn, which uses spiking neurons with localized receptive fields to compute attention while reducing memory requirements. Specifically, we introduce a LRF method into SSA to assign higher weights to neighboring regions, strengthening local modeling and improving performance. Building on this, we approximate the resulting attention computation via charge-fire-reset dynamics, eliminating explicit attention-matrix storage and reducing inference-time memory. Extensive experiments on visual tasks confirm that our method reduces memory overhead while delivering significant performance improvements. These results establish it as a key unit for achieving energy-efficient Spiking Transformers.
Paper Structure (26 sections, 2 theorems, 33 equations, 6 figures, 3 tables)

This paper contains 26 sections, 2 theorems, 33 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminary
  4. Spiking Neuron Model
  5. Spiking Self-attention Mechanisms
  6. Problem Analysis in Spiking Transformer
  7. Limited Local Modeling Capability
  8. High Memory Requirements during Inference
  9. Method
  10. Spiking Self-Attention with Local Receptive Fields
  11. Implementing Self-Attention through Neuronal Dynamics
  12. Overall Architecture
  13. Experiment
  14. Image classification & Semantic Segmentation
  15. Enhanced Local Modeling Ability with Lower Memory Requirements
  16. ...and 11 more sections

Key Result

Theorem 1

Let $i \in \mathcal{N} =\{1, \cdots, n\}$ denotes the token position and defined the Manhattan distance between two elements as $\Delta = d(i,j) = |i-j|$. The normalized attention weight of VSA is $\alpha_{ij}^{\text{vsa}} \propto \exp(-\beta \Delta)$. For SSA, the weight satisfies $\alpha_{ij}^{\te

Figures (6)

  • Figure 1: (a) Limited Local Modeling Capability: For a given n-th query (blue), VSA captures only limited and local relation. (b) High Memory Requirements: SSA requires explicit storage of their associated attention scores ($\mathbf{QK}$ or $\mathbf{KV}$), leading to substantial computational overhead.
  • Figure 2: Mismatch between VSA and SSA attention scores: (a) and (b) show the average attention scores at different Manhattan distances, with VSA demonstrating stronger local modeling capabilities. (c) and (d) illustrate the distribution of attention scores, with VSA exhibiting lower entropy.
  • Figure 3: (a) Cognitive processes in biological vision, which exhibit local receptive field properties realized through multi-dendritic neurons. (b) The proposed LRF method together with the dynamic processes of dendritic neurons. (c) The implementation of LRF-SSA and LRF-Dyn.
  • Figure 4: Visual results for image recognition and semantic segmentation. Both LRF-SSA and LRF-Dyn produce sparser attention scores and achieve finer-grained segmentation results.
  • Figure 5: (a) Visualization of effective receptive field for different methods, where both LRF-SSA and LRF-Dyn demonstrate strong locality. (b) Comparative analysis of memory usage, accuracy, and parameter efficiency. The results show that LRF-Dyn maintains performance comparable to LRF-SSA while substantially reducing memory requirements during inference.
  • ...and 1 more figures

Theorems & Definitions (2)

  • Theorem 1
  • Theorem 2