Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TIM: An Efficient Temporal Interaction Module for Spiking Transformer

Sicheng Shen, Dongcheng Zhao, Guobin Shen, Yi Zeng

TL;DR

This work identifies a limitation in Spiking Transformers: reliance on current-time information in attention leads to underutilization of temporal dynamics in neuromorphic data. It introduces the Temporal Interaction Module (TIM), a convolution-based, plug-in component that adaptively fuses historical and current information within the Spiking Self Attention computation, with minimal parameter overhead. Through extensive experiments on neuromorphic datasets (e.g., CIFAR10-DVS, N-CALTECH101, NCARS, UCF101-DVS, HMDB51-DVS, SHD), TIM delivers state-of-the-art performance and demonstrates strong temporal processing and generalizability, including compatibility with Spike-driven Transformers. The findings highlight TIM’s potential to enhance temporal cognition in SNNs while preserving efficiency, making it a practical augmentation for neuromorphic vision and action recognition tasks.

Abstract

Spiking Neural Networks (SNNs), as the third generation of neural networks, have gained prominence for their biological plausibility and computational efficiency, especially in processing diverse datasets. The integration of attention mechanisms, inspired by advancements in neural network architectures, has led to the development of Spiking Transformers. These have shown promise in enhancing SNNs' capabilities, particularly in the realms of both static and neuromorphic datasets. Despite their progress, a discernible gap exists in these systems, specifically in the Spiking Self Attention (SSA) mechanism's effectiveness in leveraging the temporal processing potential of SNNs. To address this, we introduce the Temporal Interaction Module (TIM), a novel, convolution-based enhancement designed to augment the temporal data processing abilities within SNN architectures. TIM's integration into existing SNN frameworks is seamless and efficient, requiring minimal additional parameters while significantly boosting their temporal information handling capabilities. Through rigorous experimentation, TIM has demonstrated its effectiveness in exploiting temporal information, leading to state-of-the-art performance across various neuromorphic datasets. The code is available at https://github.com/BrainCog-X/Brain-Cog/tree/main/examples/TIM.

TIM: An Efficient Temporal Interaction Module for Spiking Transformer

TL;DR

This work identifies a limitation in Spiking Transformers: reliance on current-time information in attention leads to underutilization of temporal dynamics in neuromorphic data. It introduces the Temporal Interaction Module (TIM), a convolution-based, plug-in component that adaptively fuses historical and current information within the Spiking Self Attention computation, with minimal parameter overhead. Through extensive experiments on neuromorphic datasets (e.g., CIFAR10-DVS, N-CALTECH101, NCARS, UCF101-DVS, HMDB51-DVS, SHD), TIM delivers state-of-the-art performance and demonstrates strong temporal processing and generalizability, including compatibility with Spike-driven Transformers. The findings highlight TIM’s potential to enhance temporal cognition in SNNs while preserving efficiency, making it a practical augmentation for neuromorphic vision and action recognition tasks.

Abstract

Spiking Neural Networks (SNNs), as the third generation of neural networks, have gained prominence for their biological plausibility and computational efficiency, especially in processing diverse datasets. The integration of attention mechanisms, inspired by advancements in neural network architectures, has led to the development of Spiking Transformers. These have shown promise in enhancing SNNs' capabilities, particularly in the realms of both static and neuromorphic datasets. Despite their progress, a discernible gap exists in these systems, specifically in the Spiking Self Attention (SSA) mechanism's effectiveness in leveraging the temporal processing potential of SNNs. To address this, we introduce the Temporal Interaction Module (TIM), a novel, convolution-based enhancement designed to augment the temporal data processing abilities within SNN architectures. TIM's integration into existing SNN frameworks is seamless and efficient, requiring minimal additional parameters while significantly boosting their temporal information handling capabilities. Through rigorous experimentation, TIM has demonstrated its effectiveness in exploiting temporal information, leading to state-of-the-art performance across various neuromorphic datasets. The code is available at https://github.com/BrainCog-X/Brain-Cog/tree/main/examples/TIM.
Paper Structure (22 sections, 10 equations, 4 figures, 5 tables)

This paper contains 22 sections, 10 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Advances in SNN Temporal Processing
  4. Spiking Transformer
  5. Preliminaries
  6. Spiking Neuron
  7. Spiking Self Attention Analysis
  8. Method
  9. Spikformer Backbone
  10. TIM Unit
  11. TIM Stream
  12. Experiments
  13. CIFAR10-DVS
  14. N-CALTECH101
  15. NCARS
  16. ...and 7 more sections

Figures (4)

  • Figure 1: Comprehensive diagram of Spikformer integrated with Temporal Interaction Module (TIM): Demonstrating TIM's Plug-and-Play role within the Spike Self Attention (SSA) Block of the original Spikformer structure.
  • Figure 2: (a): Overview of the Temporal Interaction Module: Adaptive Integration of Historical and Current Temporal Data. (b): The upper part of the figure describes the process of SSA, while the lower part illustrates the process after integrating TIM Stream into SSA.
  • Figure 3: Temporal enhancement validation of TIM on CIFAR10-DVS and N-CALTECH101.
  • Figure 4: Efficiency and Generalizability Validation of TIM on CIFAR10-DVS.