Spiking Neural Network as Adaptive Event Stream Slicer

TL;DR

This work proposes SpikeSlicer, a novel-designed plug-and-play event processing method capable of splitting events stream adaptively, and provides a brand-new SNN-ANN cooperation paradigm, where the SNN acts as an efficient, low-energy data processor to assist the ANN in improving downstream performance.

Abstract

Event-based cameras are attracting significant interest as they provide rich edge information, high dynamic range, and high temporal resolution. Many state-of-the-art event-based algorithms rely on splitting the events into fixed groups, resulting in the omission of crucial temporal information, particularly when dealing with diverse motion scenarios (\eg, high/low speed).In this work, we propose SpikeSlicer, a novel-designed plug-and-play event processing method capable of splitting events stream adaptively.SpikeSlicer utilizes a low-energy spiking neural network (SNN) to trigger event slicing. To guide the SNN to fire spikes at optimal time steps, we propose the Spiking Position-aware Loss (SPA-Loss) to modulate the neuron's state. Additionally, we develop a Feedback-Update training strategy that refines the slicing decisions using feedback from the downstream artificial neural network (ANN). Extensive experiments demonstrate that our method yields significant performance improvements in event-based object tracking and recognition. Notably, SpikeSlicer provides a brand-new SNN-ANN cooperation paradigm, where the SNN acts as an efficient, low-energy data processor to assist the ANN in improving downstream performance, injecting new perspectives and potential avenues of exploration. Our code is available at https://github.com/AndyCao1125/SpikeSlicer.

Figures (12)

• Figure 1: Comparison of event slicing methods. Traditional methods slice event streams based on prefixed time intervals (a) or event counts (b). In contrast, our approach (c) utilizes SNN as a dynamic event processor for adaptive event slicing. The sliced sub-event streams can be converted into various event representations with robust information and then applied to multiple downstream tasks.
• Figure 2: Overview of our method. The input events are first fed into an SNN, and the event is determined to be sliced when a spike occurs. To find the accurate slicing time, the neighborhood search method explores other time steps. and feeds event representations to the downstream ANN model (e.g., object tracker or recognizer). The ANN model then offers feedback, which guides the SNN in firing spikes at the optimal slicing time by supervising the membrane potential through the Spiking Position-aware Loss $\mathcal{L}_{SPA}$.
• Figure 3: Empirical observations: (a) Hill effect in adaptive slicing process; (b) Impact of hyperparameter $\alpha$ settings on TransT tracker TransT and (c) DiMP tracker bhat2019learning.
• Figure 4: (a) Experiments on comparing different loss functions on a simple event slicing task. Our proposed Mem-Loss and LA-Loss require only a small number of iterations to supervise the SNN to activate spikes at the desired time steps; (b) Experiments on different hyperparameter settings. Our dynamic tuning method can stably converge towards the optimal spiking time (colored in green). In contrast, using a fixed $\alpha$ results in unstable training and challenges in finding the optimal point.
• Figure 5: Visualization results on FE108 dataset. The white box denotes the zoom-in area. Our adaptive event slicing method provides better tracking performance than fixed counterparts while enabling edge enhancement (a,b) and redundancy removal (c).