Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FSTA-SNN:Frequency-based Spatial-Temporal Attention Module for Spiking Neural Networks

Kairong Yu, Tianqing Zhang, Hongwei Wang, Qi Xu

TL;DR

The paper tackles the energy inefficiency of Spiking Neural Networks by performing a frequency-domain analysis of intermediate spikes, revealing consistent spectral patterns across architectures and a depth-driven shift from vertical to horizontal feature emphasis. It introduces FSTA, a plug-and-play module composed of a DCT-based spatial attention submodule and a temporal amplitude regulation submodule, to suppress redundant spikes while boosting feature learning with minimal parameter overhead. Empirical results on CIFAR-10/100, ImageNet, and CIFAR10-DVS demonstrate state-of-the-art accuracy improvements and a substantial spike-firing-rate reduction of about 33.99%, with energy consumption kept near baseline. This work provides a practical framework for frequency-aware sparsification in SNNs and offers guidance for designing energy-efficient neuromorphic systems.

Abstract

Spiking Neural Networks (SNNs) are emerging as a promising alternative to Artificial Neural Networks (ANNs) due to their inherent energy efficiency. Owing to the inherent sparsity in spike generation within SNNs, the in-depth analysis and optimization of intermediate output spikes are often neglected. This oversight significantly restricts the inherent energy efficiency of SNNs and diminishes their advantages in spatiotemporal feature extraction, resulting in a lack of accuracy and unnecessary energy expenditure. In this work, we analyze the inherent spiking characteristics of SNNs from both temporal and spatial perspectives. In terms of spatial analysis, we find that shallow layers tend to focus on learning vertical variations, while deeper layers gradually learn horizontal variations of features. Regarding temporal analysis, we observe that there is not a significant difference in feature learning across different time steps. This suggests that increasing the time steps has limited effect on feature learning. Based on the insights derived from these analyses, we propose a Frequency-based Spatial-Temporal Attention (FSTA) module to enhance feature learning in SNNs. This module aims to improve the feature learning capabilities by suppressing redundant spike features.The experimental results indicate that the introduction of the FSTA module significantly reduces the spike firing rate of SNNs, demonstrating superior performance compared to state-of-the-art baselines across multiple datasets.

FSTA-SNN:Frequency-based Spatial-Temporal Attention Module for Spiking Neural Networks

TL;DR

The paper tackles the energy inefficiency of Spiking Neural Networks by performing a frequency-domain analysis of intermediate spikes, revealing consistent spectral patterns across architectures and a depth-driven shift from vertical to horizontal feature emphasis. It introduces FSTA, a plug-and-play module composed of a DCT-based spatial attention submodule and a temporal amplitude regulation submodule, to suppress redundant spikes while boosting feature learning with minimal parameter overhead. Empirical results on CIFAR-10/100, ImageNet, and CIFAR10-DVS demonstrate state-of-the-art accuracy improvements and a substantial spike-firing-rate reduction of about 33.99%, with energy consumption kept near baseline. This work provides a practical framework for frequency-aware sparsification in SNNs and offers guidance for designing energy-efficient neuromorphic systems.

Abstract

Spiking Neural Networks (SNNs) are emerging as a promising alternative to Artificial Neural Networks (ANNs) due to their inherent energy efficiency. Owing to the inherent sparsity in spike generation within SNNs, the in-depth analysis and optimization of intermediate output spikes are often neglected. This oversight significantly restricts the inherent energy efficiency of SNNs and diminishes their advantages in spatiotemporal feature extraction, resulting in a lack of accuracy and unnecessary energy expenditure. In this work, we analyze the inherent spiking characteristics of SNNs from both temporal and spatial perspectives. In terms of spatial analysis, we find that shallow layers tend to focus on learning vertical variations, while deeper layers gradually learn horizontal variations of features. Regarding temporal analysis, we observe that there is not a significant difference in feature learning across different time steps. This suggests that increasing the time steps has limited effect on feature learning. Based on the insights derived from these analyses, we propose a Frequency-based Spatial-Temporal Attention (FSTA) module to enhance feature learning in SNNs. This module aims to improve the feature learning capabilities by suppressing redundant spike features.The experimental results indicate that the introduction of the FSTA module significantly reduces the spike firing rate of SNNs, demonstrating superior performance compared to state-of-the-art baselines across multiple datasets.
Paper Structure (24 sections, 14 equations, 5 figures, 6 tables)

This paper contains 24 sections, 14 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Frequency Domain Learning in ANNs
  4. Frequency Applications in SNNs
  5. Attention in SNNs
  6. SNN Frequency Analysis
  7. SNN Fundamentals
  8. Frequency Fundamentals
  9. Frequency Analysis
  10. Observation 1. The structural differences and dataset variations do not affect the learning preferences of SNNs
  11. Observation 2: Increasing the time step has limited potential to enhance performance
  12. Methodology
  13. DCT-based Spatial Attention Submodule
  14. Temporal Attention Submodule for Amplitude Regulation
  15. Frequency-based Spatial-Temporal Attention Module
  16. ...and 9 more sections

Figures (5)

  • Figure 1: Comparison of the output spike frequency spectrum distribution of SNNs across different model structures, datasets, layer depths, and temporal perspectives. (a) The spectral distribution of spike outputs at different layers of the ResNet architecture at the same time step in static dataset. (b) The spectral distribution of spike outputs at different layers of the ResNet architecture at the same time step in dynamic dataset. (c) The spectral distribution of spike outputs at different layers of the VGG architecture at the same time step in static dataset. (d) The temporal variation of the frequency bands where spectral energy is most concentrated across different layers. Here, Amp, Freq, and T represent amplitude, frequency range and timesteps.
  • Figure 2: Overview of the FSTA module and its internal submodules structure
  • Figure 3: The different combinations of modules
  • Figure 4: Comparison of spike firing rates at various layers between FSTA-SNN and vanilla SNN. The legend represents the average firing rate of the entire network.
  • Figure 5: Visualization of the FSTA-SNN and vanilla SNN analysis. (a) Distribution of energy accumulation across frequency bands in the time dimension; (b) Contour plots of the averaged temporal frequency spectrum energy; (c) Grad-CAM visualization for the same sample.