SpikeVideoFormer: An Efficient Spike-Driven Video Transformer with Hamming Attention and $\mathcal{O}(T)$ Complexity

TL;DR

SpikeVideoFormer introduces an energy-efficient spike-driven video Transformer that achieves linear temporal complexity $O(T)$ through spike-driven Hamming attention (SDHA) and a joint space-time attention design. By replacing real-valued dot-product attention with normalized Hamming similarity and leveraging neuron-level temporal encoding, the approach delivers state-of-the-art performance among SNNs on video classification, human pose tracking, and semantic segmentation while substantially reducing energy consumption. The method is validated across three tasks with consistent efficiency advantages over ANN baselines, and ablations highlight the pivotal role of Hamming attention, joint space-time fusion, and proper hyperparameter choices. Overall, SpikeVideoFormer demonstrates that SNN-based Transformers can scale to video-level tasks with competitive accuracy and significant energy efficiency, suggesting broad practicality for low-power vision systems.

Abstract

Spiking Neural Networks (SNNs) have shown competitive performance to Artificial Neural Networks (ANNs) in various vision tasks, while offering superior energy efficiency. However, existing SNN-based Transformers primarily focus on single-image tasks, emphasizing spatial features while not effectively leveraging SNNs' efficiency in video-based vision tasks. In this paper, we introduce SpikeVideoFormer, an efficient spike-driven video Transformer, featuring linear temporal complexity $\mathcal{O}(T)$. Specifically, we design a spike-driven Hamming attention (SDHA) which provides a theoretically guided adaptation from traditional real-valued attention to spike-driven attention. Building on SDHA, we further analyze various spike-driven space-time attention designs and identify an optimal scheme that delivers appealing performance for video tasks, while maintaining only linear temporal complexity. The generalization ability and efficiency of our model are demonstrated across diverse downstream video tasks, including classification, human pose tracking, and semantic segmentation. Empirical results show our method achieves state-of-the-art (SOTA) performance compared to existing SNN approaches, with over 15\% improvement on the latter two tasks. Additionally, it matches the performance of recent ANN-based methods while offering significant efficiency gains, achieving $\times 16$, $\times 10$ and $\times 5$ improvements on the three tasks. https://github.com/JimmyZou/SpikeVideoFormer

Figures (10)

• Figure 1: Architecture of our proposed SpikeVideoFormer. An input video with a shape of $T\times H\times W\times 3$ undergoes temporal spiking over time, after which it passes through two spike-driven CNN blocks, along with downsample modules. This is followed by two spike-driven spatiotemporal transformers, also accompanied by downsample modules. Finally, the extracted video features are forwarded to the regression or classification head for downstream tasks.
• Figure 2: Intuitive comparison of attention scores between spike query and keys using Dot-product (Dot.) and normalized Hamming similarity (Ham.). When the spike query contains no elements, the dot-product ignores the corresponding elements in the spike keys, resulting in identical scores for four distinct spike keys as examples. This illustrates the dot-product's limitation in accurately capturing the similarity between binary spike vectors.
• Figure 3: Empirical evidence of average error $|f_{\mathcal{H}} - g(f_{\mathcal{C}})|$ with respect to embeddings size $D$.
• Figure 4: Space-time Spike-Driven Attention Designs.
• Figure 5: Architecture Details of three different spike-driven space-time attentions.

Theorems & Definitions (3)

• Proposition 3.1: JL Lemma on Binary Embedding jacques2013robustyi2015binary
• Lemma 1.1: Johnson–Lindenstrauss Lemma on Binary Embedding jacques2013robustyi2015binary
• proof