Activation-wise Propagation: A Universal Strategy to Break Timestep Constraints in Spiking Neural Networks for 3D Data Processing

TL;DR

The paper tackles the timestep constraint in spiking neural networks used for real-time 3D data processing by introducing Activation-wise Membrane Potential Propagation (AMP2). AMP2 provides a universal, architecture-agnostic mechanism that propagates membrane potentials across adjacent layers via residual connections, eliminating the need for iterative per-layer spiking updates. Empirically, AMP2 improves Spiking PointNet on ModelNet40 (87.36% to 89.74%) and can surpass ANN PointNet on the DVS128 Gesture dataset, while also benefiting PointNet and PointNet++ in 1-timestep configurations and showing favorable energy/latency characteristics. This approach broadens the applicability of SNNs to multiple 3D modalities (point clouds and event streams) without increasing timesteps, though gains are more modest for deeper networks, pointing to future work on optimizing deep SNNs.

Abstract

Due to their event-driven and parameter-efficient effect, spiking neural networks (SNNs) show potential in tasks requiring real-time multi-sensor perception, such as autonomous driving. The spiking mechanism facilitates sparse encoding, enabling spatial and temporal data to be represented in a discrete manner. However, SNNs still lag behind artificial neural networks (ANNs) in terms of performance and computational efficiency. One major challenge in SNNs is the timestep-wise iterative update of neuronal states, which makes it difficult to achieve an optimal trade-off among accuracy, latency, and training cost. Although some methods perform well with shorter timesteps, few propose strategies to overcome such constraint effectively. Moreover, many recent SNN advancements rely on either optimizations tailored to specific architectures or a collection of specialized neuron-level strategies. While these approaches can enhance performance, they often lead to increased computational expense and restrict their application to particular architectures or modalities. This leaves room for further exploration of simple, universal, and structure-agnostic strategies that could offer broader applicability and efficiency. In this paper, we introduce Activation-wise Membrane Potential Propagation (AMP2), a novel state update mechanism for spiking neurons. Inspired by skip connections in deep networks, AMP2 incorporates the membrane potential of neurons into network, eliminating the need for iterative updates. Our method achieves significant improvements across various 3D modalities, including 3D point clouds and event streams, boosting Spiking PointNet's accuracy on ModelNet40 from 87.36% to 89.74% and surpassing ANN PointNet in recognition accuracy on the DVS128 Gesture dataset.

Figures (5)

• Figure 1: Comparison between Timestep-wise Update and Activation-wise Membrane Potential Propagation (AMP2) during forward propagation. For input points of shape [B, N, C], the spiking process in timestep-wise updates requires multiple executions within each neuron layer to learn sparse representations at different positions, necessitating temporal expansion along an additional time dimension. In contrast, AMP2 links the initialization and reset processes of neuron layers, enabling the entire network to optimize a shared neural dynamic process.
• Figure 2: The workflow of Activation-wise Membrane Potential Propagation (AMP2) applied to stacked MLP blocks in Spiking PointNet. The classification network takes $b\times n$ points as input, aggregates feature through multiple MLP blocks, and outputs predictions via classification MLP. The green regions in the middle represent data transmitted in non-spiking form, while the blue background indicates spiking values. We experimented with two methods to aggregate membrane potential (MP) into the forward propagation: element-wise addition and element-wise multiplication. $mlp$ denotes the process of feature transformations, while $MLP$ denotes the reduced features for classification.
• Figure 3: Inference time (in seconds) when $B=1$, $C=3$, and $N=1024$, measured on the neuromorphic processor Lynxi HP201.
• Figure 4: Performance comparison between Residual PointNet and AMP2 PointNet during training on the ModelNet40 dataset T1 represents the timestep update network with $T=1$, while Residual-T4 refers to the Residual PointNet with $T=4$. The total number of training epochs is 200.
• Figure 5: Performance comparison during training on the ModelNet40 dataset between AMP2 PointNet without adjacent MP connections and the full AMP2 PointNet.