HDI-Former: Hybrid Dynamic Interaction ANN-SNN Transformer for Object Detection Using Frames and Events

TL;DR

HDI-Former tackles multimodal object detection from frames and events with a directly trained hybrid ANN-SNN Transformer. It couples a Semantic-Enhanced Swin Transformer (ANN branch) with a Spiking Swin Transformer (SNN branch) via a bio-inspired Dynamic Interaction mechanism, enabling effective cross-modality fusion while preserving energy efficiency. Key innovations include Semantic-Enhanced Multi-head Self-Attention with Relative Semantic Embedding for the RGB stream and a Spiking Swin Transformer with Q-K attention for events, achieving state-of-the-art accuracy with substantial energy savings on the DSEC-Detector and Gen1 datasets. The approach demonstrates strong practical impact for real-time, low-power perception in challenging conditions and provides open-source code for broader adoption.

Abstract

Combining the complementary benefits of frames and events has been widely used for object detection in challenging scenarios. However, most object detection methods use two independent Artificial Neural Network (ANN) branches, limiting cross-modality information interaction across the two visual streams and encountering challenges in extracting temporal cues from event streams with low power consumption. To address these challenges, we propose HDI-Former, a Hybrid Dynamic Interaction ANN-SNN Transformer, marking the first trial to design a directly trained hybrid ANN-SNN architecture for high-accuracy and energy-efficient object detection using frames and events. Technically, we first present a novel semantic-enhanced self-attention mechanism that strengthens the correlation between image encoding tokens within the ANN Transformer branch for better performance. Then, we design a Spiking Swin Transformer branch to model temporal cues from event streams with low power consumption. Finally, we propose a bio-inspired dynamic interaction mechanism between ANN and SNN sub-networks for cross-modality information interaction. The results demonstrate that our HDI-Former outperforms eleven state-of-the-art methods and our four baselines by a large margin. Our SNN branch also shows comparable performance to the ANN with the same architecture while consuming 10.57$\times$ less energy on the DSEC-Detection dataset. Our open-source code is available in the supplementary material.

Figures (11)

• Figure 1: Comparison of our HDI-Former with state-of-the-art methods on the DSEC-Detection dataset. Our HDI-Former achieves a significant mAP improvement over unimodal methods while keeping comparable energy consumption, and it outperforms multimodal methods while reducing energy consumption.
• Figure 2: The pipeline of Hybrid Dynamic Interaction ANN-SNN Transformer (HDI-Former). (a) Our HDI-Former consists of two sub-networks and connects an ANN and an SNN via dynamic interaction. The ANN branch processes frames using Semantic-Enhanced Swin Transformer blocks, while the SNN branch handles events with Swin QKFormer and Swin Spikformer blocks. Finally, an asynchronous fusion module li2023sodformer combines two streams to predict the detection results. (b) A bio-inspired Dynamic Interaction mechanism between ANN and SNN sub-networks to achieve cross-modality interaction and leverage the complementarity of two streams.
• Figure 3: The spiked self-attention with shifted windows and two kinds of blocks in the Spiking Swin Transformer.
• Figure 4: Representative examples of different object detection results in various scenarios on the DSEC-Detection dataset Gehrig24nature.
• Figure 5: Representative visualization results of our SNN-branch and EMS-YOLO on the Gen1 dataset de2020large.