SMamba: Sparse Mamba for Event-based Object Detection

TL;DR

SMamba addresses the high computation of transformer-based event detectors by introducing adaptive sparsification that preserves global modeling. It combines STCA to quantify token informativeness, IPL-Scan to prioritize high-information tokens for efficient local interactions, and GCI to enable 3D global channel interactions, forming a four-stage architecture with SSM and SCMM blocks. The approach yields superior accuracy with reduced FLOPs and parameters across Gen1, 1Mpx, and eTram, demonstrating robust, scene-adaptive performance. This work advances practical, real-time event-based detection by balancing global context with sparsity-aware processing, facilitating deployment in autonomous driving and surveillance scenarios.

Abstract

Transformer-based methods have achieved remarkable performance in event-based object detection, owing to the global modeling ability. However, they neglect the influence of non-event and noisy regions and process them uniformly, leading to high computational overhead. To mitigate computation cost, some researchers propose window attention based sparsification strategies to discard unimportant regions, which sacrifices the global modeling ability and results in suboptimal performance. To achieve better trade-off between accuracy and efficiency, we propose Sparse Mamba (SMamba), which performs adaptive sparsification to reduce computational effort while maintaining global modeling capability. Specifically, a Spatio-Temporal Continuity Assessment module is proposed to measure the information content of tokens and discard uninformative ones by leveraging the spatiotemporal distribution differences between activity and noise events. Based on the assessment results, an Information-Prioritized Local Scan strategy is designed to shorten the scan distance between high-information tokens, facilitating interactions among them in the spatial dimension. Furthermore, to extend the global interaction from 2D space to 3D representations, a Global Channel Interaction module is proposed to aggregate channel information from a global spatial perspective. Results on three datasets (Gen1, 1Mpx, and eTram) demonstrate that our model outperforms other methods in both performance and efficiency.

Figures (6)

• Figure 1: (a) The event sparsity statistics on three datasets. Each point represents a scenario. (b) Compared with calculating all regions uniformly in Transformer, our proposed SMamba can suppress non-event and noisy regions (gray regions) from participating in calculations and only retain information-rich areas (yellow regions), alleviating computational overhead significantly and suppressing noise disturbance, simultaneously. (c) mAP-FLOPs-Parameters comparison between state-of-the-art methods and our SMamba on Gen1, where the circle radius is the parameter. SMamba achieves a superior balance between accuracy and efficiency.
• Figure 2: The architecture of SMamba. Given the input event stream, the STCA module (as illustrated in (d)) evaluates the information content of tokens based on spatiotemporal continuity and generates a sparsification map to guide the sparsification operation. Simultaneously, the event stream is preprocessed and transferred to four stages for comprehensive global interactions across spatial and channel dimensions. The first two stages employ the Sparse Spatial Mamba (SSM) layer (as detailed in (b)) to facilitate global spatial interactions on kept tokens. The final two stages utilize the Spatial-Channel Mixing Mamba (SCMM) layer (as depicted in (c)) to permute global modeling within the 3D representation space.
• Figure 3: Sparse SS2D. For ease of observation, a large patch size is utilized for tokenization. The uninformative and noisy tokens (gray regions) are discarded from the calculation and the kept tokens (yellow regions) are expanded into three scanning sequences by Bidi-Scan and IPL-Scan, each sequence is processed in parallel using separate S6 blocks. The window sorting results of IPL-Scan are indicated at the center of each window (highlighted in red).
• Figure 4: Bidirectional Channel Scan. The feature map is unfolded along the spatial dimension and arranged into two sequences along the channel direction, which are then input into S6 for full spatial-level channel information interaction.
• Figure 5: Visualizations of raw events, score map, sparsification map and sparsification results.