Efficient 3D Recognition with Event-driven Spike Sparse Convolution

TL;DR

This work tackles the high energy and computation demands of 3D recognition by leveraging spikes in sparse 3D data. It introduces Spike Voxel Coding (SVC) to voxelize point clouds into sparse spiking representations and Spike Sparse Convolution (SSC) to compute features only at active spike locations, forming the E-3DSNN backbone with residual membrane potentials. The approach yields state-of-the-art efficiency and competitive accuracy across 3D classification, detection, and segmentation on ModelNet40, KITTI, and Semantic KITTI, with significantly reduced energy consumption. The results demonstrate the practical viability of neuromorphic hardware-friendly SNNs for diverse 3D vision tasks, including on large-scale NuScenes data, signaling a strong potential for low-power autonomous systems.

Abstract

Spiking Neural Networks (SNNs) provide an energy-efficient way to extract 3D spatio-temporal features. Point clouds are sparse 3D spatial data, which suggests that SNNs should be well-suited for processing them. However, when applying SNNs to point clouds, they often exhibit limited performance and fewer application scenarios. We attribute this to inappropriate preprocessing and feature extraction methods. To address this issue, we first introduce the Spike Voxel Coding (SVC) scheme, which encodes the 3D point clouds into a sparse spike train space, reducing the storage requirements and saving time on point cloud preprocessing. Then, we propose a Spike Sparse Convolution (SSC) model for efficiently extracting 3D sparse point cloud features. Combining SVC and SSC, we design an efficient 3D SNN backbone (E-3DSNN), which is friendly with neuromorphic hardware. For instance, SSC can be implemented on neuromorphic chips with only minor modifications to the addressing function of vanilla spike convolution. Experiments on ModelNet40, KITTI, and Semantic KITTI datasets demonstrate that E-3DSNN achieves state-of-the-art (SOTA) results with remarkable efficiency. Notably, our E-3DSNN (1.87M) obtained 91.7\% top-1 accuracy on ModelNet40, surpassing the current best SNN baselines (14.3M) by 3.0\%. To our best knowledge, it is the first direct training 3D SNN backbone that can simultaneously handle various 3D computer vision tasks (e.g., classification, detection, and segmentation) with an event-driven nature. Code is available: https://github.com/bollossom/E-3DSNN/.

Figures (6)

• Figure 1: The workflow of our efficient 3D SNN backbone (E-3DSNN), which uses residual connections between membrane potentials and handles various 3D computer vision tasks with only sparse ACcumulate. It consists of two main components: the Spike Voxel Coding (SSC) and Spike Sparse Convolution (SSC). The SVC scheme first voxelizes the input 3D points. Then, the voxelized data is transformed into spatio-temporal spike trains using sparse convolution and spiking neurons. The SSC block only calculates the overlapping activation features between the center of the point cloud and the convolution kernel.
• Figure 2: Comparison of different point cloud pre-processing ways in SNN. (a) The vanilla point-based method lan2023efficientren2024spikingwu2024spikepoint directly processes raw points, but the inherent sparsity of SNNs can obscure local geometric details. (b) We proposed a spike voxel coding (SVC) scheme, which leverages the sparsity of SNNs and, after additional voxelization pretreatment, can handle structural data with higher efficiency and lower power consumption.
• Figure 3: Comparison of Spike Sparse Conv (SSC) and Vanilla Spike Conv (VSC). Inputs and outputs are shown as 2D features for simplicity: green for activation, red for spikes, and white for no activation. On a neuromorphic chip, when a spike occurs, the address mapping function finds the synapses and neurons that need to be added and then takes out the corresponding weights to perform the addition operations. The only difference between VSC and SSC is the addressing mapping function. In SSC, it is specified that the convolution is performed only if there is a spike input at the position corresponding to $W_4$ (the center position of the convolution kernel). VSC does not have this restriction.
• Figure 4: Visualization of E-3DSNN in hidden layer features and results. (a) We compared the hidden layer features generated with (top) and without SVC and SSC (bottom). (b) We compared the results of our E-3DSNN (top) in detection and segmentation with the ground truth (bottom).
• Figure 5: Visualization of results on KITTI dataset KITTI. Our E-3DSNN excels in the 3D object detection task