Hybrid Spiking Vision Transformer for Object Detection with Event Cameras

TL;DR

The paper addresses object detection from event camera data by bridging Transformer-based ANNs with Spiking Neural Networks to exploit high temporal resolution and low-power operation. The proposed Hybrid Spiking Vision Transformer (HsVT) uses four blocks that fuse spatial feature extraction (MaxViT with Block-SA and Grid-SA, via SpikingMLP) and temporal feature extraction (LSTM in early blocks and STFE in the final block) within a time-propagation framework across time steps. A new Fall Detection Dataset based on event streams is introduced, alongside extensive experiments on GEN1, Fall Detection, and Aircraft datasets, including ablation studies to justify component choices and placement. The results demonstrate competitive mAP with fewer parameters than baselines and provide energy-consumption estimates, highlighting the practical potential of hybrid ANN-SNN designs for efficient, privacy-preserving event-based detection in real-world scenarios.

Abstract

Event-based object detection has gained increasing attention due to its advantages such as high temporal resolution, wide dynamic range, and asynchronous address-event representation. Leveraging these advantages, Spiking Neural Networks (SNNs) have emerged as a promising approach, offering low energy consumption and rich spatiotemporal dynamics. To further enhance the performance of event-based object detection, this study proposes a novel hybrid spike vision Transformer (HsVT) model. The HsVT model integrates a spatial feature extraction module to capture local and global features, and a temporal feature extraction module to model time dependencies and long-term patterns in event sequences. This combination enables HsVT to capture spatiotemporal features, improving its capability to handle complex event-based object detection tasks. To support research in this area, we developed and publicly released The Fall Detection Dataset as a benchmark for event-based object detection tasks. This dataset, captured using an event-based camera, ensures facial privacy protection and reduces memory usage due to the event representation format. We evaluated the HsVT model on GEN1 and Fall Detection datasets across various model sizes. Experimental results demonstrate that HsVT achieves significant performance improvements in event detection with fewer parameters.

Figures (7)

• Figure 1: Event Stream Representation with Time Intervals. Each vertical line represents an event occurrence, while equidistant red lines represent the time intervals. Each event is represented by a triplet $(t, \langle x, y \rangle, p)$, denoting its spatial and temporal coordinates.
• Figure 2: Architecture of the HsVT Network. The architecture of the proposed HsVT network consists of four Blocks, each incorporating spatial feature extraction and temporal feature extraction components.
• Figure 3: Spatial Feature Extraction in the HsVT Architecture. This figure illustrates the process of spatial feature extraction within the HsVT architecture. The components, including Block-SA, SpikingMLP, Grid-SA, and a second SpikingMLP, collaboratively work to extract spatial features from the input data.
• Figure 4: Visual comparison of attention patterns from Block-SA and Grid-SA. (a) Block-SA captures local fine-grained spatial correlations. (b) Grid-SA highlights broader, long-range dependencies. (c) The overlay illustrates how these mechanisms complement each other in spatial understanding.
• Figure 5: Spiking Temporal Feature Extraction in the HsVT Architecture. This figure illustrates the process of temporal feature extraction within the HsVT architecture, where the first three blocks employ the LSTM model and the final block utilizes the STFE model. This architecture integrates these models to capture the temporal dependencies in event data, facilitating the extraction of rich temporal features crucial for event-based object detection tasks.