Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DTA: Dual Temporal-channel-wise Attention for Spiking Neural Networks

Minje Kim, Minjun Kim, Xu Yang

TL;DR

This work addresses the challenge of effectively leveraging temporal information in Spiking Neural Networks (SNNs) by introducing Dual Temporal-channel-wise Attention (DTA), which combines Temporal-channel-wise identical Cross Attention (T-XA) and Temporal-channel-wise Non-identical Attention (T-NA) in a single DTA block. By embedding a DTA block into an MS-ResNet backbone, the authors achieve state-of-the-art performance on static and dynamic datasets (CIFAR10/100, ImageNet-1k, CIFAR10-DVS) with fewer time steps, demonstrating improved spike representations and temporal-channel modeling. The contributions include the first integration of both identical and non-identical attention mechanisms for temporal-channel processing in SNNs, detailed designs for T-XA and T-NA (including LTCA/GTCA components), and comprehensive ablations and visual analyses supporting the effectiveness and efficiency of a single-block attention approach. The work offers a practical pathway to more energy-efficient SNNs with enhanced temporal dynamics, aided by publicly released code.

Abstract

Spiking Neural Networks (SNNs) present a more energy-efficient alternative to Artificial Neural Networks (ANNs) by harnessing spatio-temporal dynamics and event-driven spikes. Effective utilization of temporal information is crucial for SNNs, leading to the exploration of attention mechanisms to enhance this capability. Conventional attention operations either apply identical operation or employ non-identical operations across target dimensions. We identify that these approaches provide distinct perspectives on temporal information. To leverage the strengths of both operations, we propose a novel Dual Temporal-channel-wise Attention (DTA) mechanism that integrates both identical/non-identical attention strategies. To the best of our knowledge, this is the first attempt to concentrate on both the correlation and dependency of temporal-channel using both identical and non-identical attention operations. Experimental results demonstrate that the DTA mechanism achieves state-of-the-art performance on both static datasets (CIFAR10, CIFAR100, ImageNet-1k) and dynamic dataset (CIFAR10-DVS), elevating spike representation and capturing complex temporal-channel relationship. We open-source our code: https://github.com/MnJnKIM/DTA-SNN.

DTA: Dual Temporal-channel-wise Attention for Spiking Neural Networks

TL;DR

This work addresses the challenge of effectively leveraging temporal information in Spiking Neural Networks (SNNs) by introducing Dual Temporal-channel-wise Attention (DTA), which combines Temporal-channel-wise identical Cross Attention (T-XA) and Temporal-channel-wise Non-identical Attention (T-NA) in a single DTA block. By embedding a DTA block into an MS-ResNet backbone, the authors achieve state-of-the-art performance on static and dynamic datasets (CIFAR10/100, ImageNet-1k, CIFAR10-DVS) with fewer time steps, demonstrating improved spike representations and temporal-channel modeling. The contributions include the first integration of both identical and non-identical attention mechanisms for temporal-channel processing in SNNs, detailed designs for T-XA and T-NA (including LTCA/GTCA components), and comprehensive ablations and visual analyses supporting the effectiveness and efficiency of a single-block attention approach. The work offers a practical pathway to more energy-efficient SNNs with enhanced temporal dynamics, aided by publicly released code.

Abstract

Spiking Neural Networks (SNNs) present a more energy-efficient alternative to Artificial Neural Networks (ANNs) by harnessing spatio-temporal dynamics and event-driven spikes. Effective utilization of temporal information is crucial for SNNs, leading to the exploration of attention mechanisms to enhance this capability. Conventional attention operations either apply identical operation or employ non-identical operations across target dimensions. We identify that these approaches provide distinct perspectives on temporal information. To leverage the strengths of both operations, we propose a novel Dual Temporal-channel-wise Attention (DTA) mechanism that integrates both identical/non-identical attention strategies. To the best of our knowledge, this is the first attempt to concentrate on both the correlation and dependency of temporal-channel using both identical and non-identical attention operations. Experimental results demonstrate that the DTA mechanism achieves state-of-the-art performance on both static datasets (CIFAR10, CIFAR100, ImageNet-1k) and dynamic dataset (CIFAR10-DVS), elevating spike representation and capturing complex temporal-channel relationship. We open-source our code: https://github.com/MnJnKIM/DTA-SNN.

Paper Structure

This paper contains 14 sections, 14 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. SNN Training Methods
  4. Attention Mechanism in SNNs
  5. Methods
  6. Neuron Model and Surrogate Gradient for SNNs
  7. Dual Temporal-channel-wise Attention
  8. Temporal-channel-wise identical Cross Attention
  9. Temporal-channel-wise Non-identical Attention
  10. Experiments
  11. Settings
  12. Comparisons with SOTA methods
  13. Ablation Study
  14. Conclusion

Figures (3)

  • Figure 1: (a) shows the overall workflow of our proposed Dual Temporal-channel-wise Attention (DTA) mechanism. The DTA incorporates Temporal-channel-wise identical Cross Attention (T-XA) and Temporal-channel-wise Non-identical Attention (T-NA) modules to administer an identical operation and non-identical operations across temporal and channel dimensions, respectively. (b) illustrates our proposed DTA-SNN architecture, which embeds a single DTA block into a spiking neural network (SNN).
  • Figure 2: (a) presents the overall structure of the T-XA module, (b) illustrates the design of the T-NA module, and (c) shows the overall architecture of the Temporal Local Attention (TLA), the Channel Local Attention (CLA), the Local Temporal-Channel Attention (LTCA), and the Global Temporal-Channel Attention (GTCA).
  • Figure 3: Grad-CAM results for CIFAR10-DVS, visualized across 10 classes in four rows. The first row shows the original images, followed by Grad-CAM visualizations generated using only T-XA in the second row and only T-NA in the third row. The fourth row illustrates the results of the DTA mechanism.