Fire on Motion: Optimizing Video Pass-bands for Efficient Spiking Action Recognition

TL;DR

The paper identifies a fundamental pass-band mismatch in spiking neural networks for video, showing that the LIF neuron inherently acts as a temporal low-pass that underutilizes motion information. It introduces Pass-Band Optimizer (PBO), a lightweight, plug-in pre-filter with a two-tap structure and learnable parameters that reshapes the temporal frequency content fed into the membrane, aligning it with task-relevant motion signals. Across RGB action recognition and RGB+DVS weakly supervised video anomaly detection, PBO yields large accuracy gains (e.g., >10 percentage points on UCF101) while keeping computational overhead minimal and requiring no backbone changes. This frequency-domain perspective offers a practical pathway to more effective and energy-efficient SNN-based video understanding.

Abstract

Spiking neural networks (SNNs) have gained traction in vision due to their energy efficiency, bio-plausibility, and inherent temporal processing. Yet, despite this temporal capacity, most progress concentrates on static image benchmarks, and SNNs still underperform on dynamic video tasks compared to artificial neural networks (ANNs). In this work, we diagnose a fundamental pass-band mismatch: Standard spiking dynamics behave as a temporal low pass that emphasizes static content while attenuating motion bearing bands, where task relevant information concentrates in dynamic tasks. This phenomenon explains why SNNs can approach ANNs on static tasks yet fall behind on tasks that demand richer temporal understanding.To remedy this, we propose the Pass-Bands Optimizer (PBO), a plug-and-play module that optimizes the temporal pass-band toward task-relevant motion bands. PBO introduces only two learnable parameters, and a lightweight consistency constraint that preserves semantics and boundaries, incurring negligible computational overhead and requires no architectural changes. PBO deliberately suppresses static components that contribute little to discrimination, effectively high passing the stream so that spiking activity concentrates on motion bearing content. On UCF101, PBO yields over ten percentage points improvement. On more complex multi-modal action recognition and weakly supervised video anomaly detection, PBO delivers consistent and significant gains, offering a new perspective for SNN based video processing and understanding.

Figures (4)

• Figure 1: Temporal power spectra computed over the full UCF101 dataset ucf101 and the effects of different filters. (a) The LIF dynamics act as a low-pass filter, suppressing high-frequency components. (b) Cascading a temporal high-pass with the LIF stage retains high-frequency content but eliminates low-frequency energy, nulling the DC component. (c) Our plug-and-play PBO module with LIF adaptively optimizes the temporal pass-band, yielding a learned trade-off between low-frequency content and motion cues.
• Figure 2: UCF101 results with a spike-driven transformer sd-transformer. (a) Top-1 accuracy vs. epoch for three schemes: LIF only (low-pass), High-pass $\rightarrow$ LIF (coarse band-pass), and PBO $\rightarrow$ LIF (ours). (b) Corresponding validation loss. (c) Layer-step spike-ratio surfaces of the LIF across training. Left$\rightarrow$right: initial, epoch 30, final. Top row: LIF only, bottom row: PBO$\rightarrow$LIF. Color encodes spike ratio in $[0,0.5]$. Red dots indicate the firing ratio used for query mapping.
• Figure 3: Anomaly scores of our methods on Color-Event UCF-Crime. Pink areas indicate the manually labelled abnormal events, purple lines represent the anomaly score and red boxes point out abnormal events on the screen.
• Figure 4: Visualization of the learned temporal mixing weights$\lambda[t]$ under different leak factors $\tau$. From left to right: initialization (epoch 0), and results at epoch 499 for $\tau\!=\!0.3,\,0.5,\,0.7,\,0.9$.