SpikePool: Event-driven Spiking Transformer with Pooling Attention

TL;DR

This work reveals that spiking transformers inherently behave as high-pass filters when processing event-based data, in contrast to Vision Transformers which act as low-pass filters. Leveraging this insight, the authors introduce SpikePool, which replaces self-attention with max-pooling-based pooling attention to yield a balanced band-pass frequency response that preserves meaningful high-frequency content while suppressing noise. Empirically, SpikePool achieves competitive classification and object-detection performance on neuromorphic datasets while delivering significant reductions in training and inference time, up to 42.5% and 32.8% respectively. The study also demonstrates the method’s generalizability across related spiking architectures, providing a principled frequency-domain framework to guide efficient neuromorphic transformer design for event-based vision.

Abstract

Building on the success of transformers, Spiking Neural Networks (SNNs) have increasingly been integrated with transformer architectures, leading to spiking transformers that demonstrate promising performance on event-based vision tasks. However, despite these empirical successes, there remains limited understanding of how spiking transformers fundamentally process event-based data. Current approaches primarily focus on architectural modifications without analyzing the underlying signal processing characteristics. In this work, we analyze spiking transformers through the frequency spectrum domain and discover that they behave as high-pass filters, contrasting with Vision Transformers (ViTs) that act as low-pass filters. This frequency domain analysis reveals why certain designs work well for event-based data, which contains valuable high-frequency information but is also sparse and noisy. Based on this observation, we propose SpikePool, which replaces spike-based self-attention with max pooling attention, a low-pass filtering operation, to create a selective band-pass filtering effect. This design preserves meaningful high-frequency content while capturing critical features and suppressing noise, achieving a better balance for event-based data processing. Our approach demonstrates competitive results on event-based datasets for both classification and object detection tasks while significantly reducing training and inference time by up to 42.5% and 32.8%, respectively.

Figures (9)

• Figure 1: Frequency analysis and performance comparison of spiking transformers. (a) Relative Log Amplitude (RLA) across transformer layers, showing contrasting frequency filtering behaviors between spiking transformer and SpikePool. Layer 1 is embedding, layers 2 and 4 are attention (star-shaped), and layers 3 and 5 are Multi-Layer Perceptron (MLP) layers. (b) Accuracy and number of parameter comparison on the N-Caltech101 dataset.
• Figure 2: Frequency domain analysis of transformer architectures. (a) Relative Log Amplitude (RLA) across normalized layer depth, showing contrasting frequency filtering behaviors between ViT-Base (low-pass) and spiking transformer (high-pass). (b) Frequency-specific noise perturbation process using Fourier domain masking. (c) Robustness comparison between ViT-Base and spiking transformer.
• Figure 3: Overall scheme of SpikePool architecture. The pooling attention replaces prior spiking self-attention block with pooling attention. Conv, MLP, and BN represent convolution, multi-layer perception, and batch normalization.
• Figure 4: Attention feature maps comparison. Input event data (top), spike-driven transformer yao2023spike output (middle), and SpikePool output (bottom).
• Figure 5: Training and inference time comparison on N-Caltech101 and Gen1 datasets. (a) N-Caltech101 showing training time per iteration for Spike-driven Transformer (SDT), SpikePool-B (SP-B), and SpikePool-S (SP-S). (b) Gen1 dataset comparing training and inference times for SpikingViT (S-ViT) and SpikePool (SP).