Fully Spiking Neural Networks for Unified Frame-Event Object Tracking

TL;DR

SpikeFET presents the first fully spiking neural network framework for unified frame-event object tracking, pairing frame-based appearance with event-driven dynamics. It introduces a Randomized Patchwork Module (RPM) to mitigate padding-induced translation bias and a Spatial-Temporal Regularization (STR) to preserve cross-temporal similarity in latent space, enabling robust cross-modal fusion via CSWin-SSA transformers. The approach demonstrates strong tracking accuracy across FE108, VisEvent, and COESOT benchmarks while delivering substantial energy efficiency relative to ANN-based and other SNN trackers. These advances suggest significant potential for low-power, edge-friendly visual tracking in challenging conditions such as low light and high-speed motion.

Abstract

The integration of image and event streams offers a promising approach for achieving robust visual object tracking in complex environments. However, current fusion methods achieve high performance at the cost of significant computational overhead and struggle to efficiently extract the sparse, asynchronous information from event streams, failing to leverage the energy-efficient advantages of event-driven spiking paradigms. To address this challenge, we propose the first fully Spiking Frame-Event Tracking framework called SpikeFET. This network achieves synergistic integration of convolutional local feature extraction and Transformer-based global modeling within the spiking paradigm, effectively fusing frame and event data. To overcome the degradation of translation invariance caused by convolutional padding, we introduce a Random Patchwork Module (RPM) that eliminates positional bias through randomized spatial reorganization and learnable type encoding while preserving residual structures. Furthermore, we propose a Spatial-Temporal Regularization (STR) strategy that overcomes similarity metric degradation from asymmetric features by enforcing spatio-temporal consistency among temporal template features in latent space. Extensive experiments across multiple benchmarks demonstrate that the proposed framework achieves superior tracking accuracy over existing methods while significantly reducing power consumption, attaining an optimal balance between performance and efficiency.

Figures (8)

• Figure 1: SpikeFET versus other tracking methods on COESOT.
• Figure 2: Visualization results of SpikeFET-Base in low light and overexposure scenarios.
• Figure 3: The overview of SpikeFET. Our SpikeFET consists of Input Processing, Feature Extraction, Feature Fusion and Tracking. Where the Event Stream is simply transformed into event temporal frames using fan2024dense. Feature Extraction consists of DownSampling and ConvFormer Spike Block cascades. Feature Fusion consists of TransFormer Spike Block cascades. Tracking uses SNN Tracking Head, and Spiking Neuron use the Spike Firing Approximation (SFA) yao2025scaling
• Figure 4: Detailed design of proposed RPM. Randomly combine template frames $\text{Z}_1$ and $\text{Z}_2$ with search frame $\text{X}$.
• Figure 5: Illusion of the proposed STR. Applying similarity loss regularization to two temporally adjacent template frames.