EAS-SNN: End-to-End Adaptive Sampling and Representation for Event-based Detection with Recurrent Spiking Neural Networks

TL;DR

The paper addresses efficient, end-to-end learning for event-based detection by introducing EAS-SNN, a framework that embeds a learnable adaptive sampling module based on recurrent convolutional SNNs. Key contributions include ARSNN (adaptive sampling), Residual Potential Dropout (RPD), and Spike-Aware Training (SAT), enabling differentiable optimization and improved sampling timing. Empirical results on neuromorphic datasets show state-of-the-art performance for spike-based methods with significantly fewer parameters and energy consumption, plus competitive gains on dense models. The approach potentially enables energy-efficient, real-time object detection on neuromorphic hardware, and demonstrates generality beyond SNNs to conventional networks.

Abstract

Event cameras, with their high dynamic range and temporal resolution, are ideally suited for object detection, especially under scenarios with motion blur and challenging lighting conditions. However, while most existing approaches prioritize optimizing spatiotemporal representations with advanced detection backbones and early aggregation functions, the crucial issue of adaptive event sampling remains largely unaddressed. Spiking Neural Networks (SNNs), which operate on an event-driven paradigm through sparse spike communication, emerge as a natural fit for addressing this challenge. In this study, we discover that the neural dynamics of spiking neurons align closely with the behavior of an ideal temporal event sampler. Motivated by this insight, we propose a novel adaptive sampling module that leverages recurrent convolutional SNNs enhanced with temporal memory, facilitating a fully end-to-end learnable framework for event-based detection. Additionally, we introduce Residual Potential Dropout (RPD) and Spike-Aware Training (SAT) to regulate potential distribution and address performance degradation encountered in spike-based sampling modules. Empirical evaluation on neuromorphic detection datasets demonstrates that our approach outperforms existing state-of-the-art spike-based methods with significantly fewer parameters and time steps. For instance, our method yields a 4.4\% mAP improvement on the Gen1 dataset, while requiring 38\% fewer parameters and only three time steps. Moreover, the applicability and effectiveness of our adaptive sampling methodology extend beyond SNNs, as demonstrated through further validation on conventional non-spiking models. Code is available at https://github.com/Windere/EAS-SNN.

Figures (6)

• Figure 1: Left: Implementing the end-to-end optimization for event-based sampling via recurrent SNNs. The typical rule-based methods truncate backpropagation, whereas our adaptive sampling module ensures continuous gradient flow. Right: The variability and heterogeneity in sampling time and count among different spiking neurons.
• Figure 2: Illustration of the overall pipeline. The adaptive sampling module, augmented with recurrent synaptic connections for improved memory mechanism, bridges the input event stream with the downstream spike-based detector. Within the unrolled computation graph, internal neuron dynamics are indicated by solid black lines, while blue and red solid lines depict the feedforward and recurrent pathways, respectively.
• Figure 3: Visualizing the detection results and corresponding embedding distributions of different models in ablation study. The red dashed line highlights the ground truth.
• Figure 4: (a) Assessing the scalability of the early aggregation step $T_m$ in mismatched testing scenarios on Gen1. (b) Evaluating the effectiveness of the proposed adaptive sampling mechanism within conventional dense neural networks on N-Caltech 101.
• Figure 5: Exploring the behavior of sampling module. The first column illustrates bounding boxes of prediction, whereas the second column provides pixel-wise firing count of spike neurons. The third and fourth columns, respectively, depict the distribution of neuron firing time and the sampling duration, quantified as the inter-spike interval.