Unveiling the Spatial-temporal Effective Receptive Fields of Spiking Neural Networks

TL;DR

This paper addresses the challenge of modeling long-range spatio-temporal dependencies in visual tasks with Transformer-based SNNs by introducing the Spatial-Temporal Effective Receptive Field (ST-ERF). It analyzes existing SNN architectures and identifies a global ST-ERF deficiency caused by locality-biased channel mixers, then proposes two channel-mixer designs, MLPixer and SRB, to enlarge the ST-ERF across timesteps. Through theoretical ST-ERF formulation and loss-based gradient analysis, the authors validate that the proposed mixers produce more global receptive fields and deliver improved performance on COCO and ADE20K, as well as on event-based tracking benchmarks. The ST-ERF framework and these architectural innovations offer practical guidance for designing SNNs capable of robust long-range spatio-temporal modeling with potential broad applicability beyond the reported tasks.

Abstract

Spiking Neural Networks (SNNs) demonstrate significant potential for energy-efficient neuromorphic computing through an event-driven paradigm. While training methods and computational models have greatly advanced, SNNs struggle to achieve competitive performance in visual long-sequence modeling tasks. In artificial neural networks, the effective receptive field (ERF) serves as a valuable tool for analyzing feature extraction capabilities in visual long-sequence modeling. Inspired by this, we introduce the Spatio-Temporal Effective Receptive Field (ST-ERF) to analyze the ERF distributions across various Transformer-based SNNs. Based on the proposed ST-ERF, we reveal that these models suffer from establishing a robust global ST-ERF, thereby limiting their visual feature modeling capabilities. To overcome this issue, we propose two novel channel-mixer architectures: \underline{m}ulti-\underline{l}ayer-\underline{p}erceptron-based m\underline{ixer} (MLPixer) and \underline{s}plash-and-\underline{r}econstruct \underline{b}lock (SRB). These architectures enhance global spatial ERF through all timesteps in early network stages of Transformer-based SNNs, improving performance on challenging visual long-sequence modeling tasks. Extensive experiments conducted on the Meta-SDT variants and across object detection and semantic segmentation tasks further validate the effectiveness of our proposed method. Beyond these specific applications, we believe the proposed ST-ERF framework can provide valuable insights for designing and optimizing SNN architectures across a broader range of tasks. The code is available at \href{https://github.com/EricZhang1412/Spatial-temporal-ERF}{\faGithub~EricZhang1412/Spatial-temporal-ERF}.

Figures (5)

• Figure 1: (a) The human visual field. (b) ERF in ANNs. (c) ST-ERF in SNNs. It extends the ERF to the temporal dimension, thus facilitating a comprehensive analysis of feature extraction in SNNs.
• Figure 2: Comparison of spatial ERF with ANN Vision Transformers and ST-ERF with different Transformer-based SNNs. (a) ViT-like architecture comparison group: ViT-B and SDT-V1. (b) Meta-architecture comparison group: CAFormer-s18 and its counterpart Meta-SDT-S.
• Figure 3: Comparison between the original channel mixer design and our proposed methods, along with their ST-ERF. For clearer visualization, the ST-ERF is averaged over timesteps. (a) Vanilla convolution-based channel mixer. (b) Proposed MLPixer architecture. (c) Proposed SRB architecture. Obviously, the vanilla convolution-based channel mixer exhibits a limited ST-ERF, whereas our MLPixer and SRB modules achieve a more global ST-ERF. Moreover, due to the reduced use of convolutions, MLPixer exhibits an even broader effective receptive field.
• Figure 4: The overall architecture of Meta-SDT, which typically comprises four hierarchical stages. The first two stages use convolution-based SNN blocks, while the latter two adopt Transformer-SNN blocks. To strengthen the global modeling capacity of SNNs, we introduce two novel channel mixer architectures, MLPixer and SRB, to replace the convolution-based SNN blocks in the first two stages.
• Figure 5: Comparison of temporal-averaged spatial ERF evolution across stages. From top to bottom are Stage 1 through Stage 4. Spikformer shows diffuse receptive fields across all stages. SDT-V1, Meta-SDT, and QKFormer exhibit more centered spatial distributions that gradually expand as depth increases. Our two Meta-SDT variants establish global spatial receptive fields in the early stages.

Theorems & Definitions (1)

• proof