Autonomous Driving with Spiking Neural Networks

TL;DR

This work introduces Spiking Autonomous Driving (SAD), the first end-to-end Spiking Neural Network that unifies perception, prediction, and planning for autonomous driving under energy constraints. SAD processes multi-view camera data to build a spatiotemporal BEV, uses a dual-pathway, spike-based prediction to forecast future states with uncertainty, and refines safe trajectories via a planning module enhanced by an SGRU, all trained with a joint loss. The approach demonstrates competitive performance on nuScenes across perception, prediction, and planning tasks while achieving substantial energy efficiency through event-driven spiking computation. This work highlights the viability and practical impact of neuromorphic, low-power architectures for safety-critical, real-world autonomous driving applications.

Abstract

Autonomous driving demands an integrated approach that encompasses perception, prediction, and planning, all while operating under strict energy constraints to enhance scalability and environmental sustainability. We present Spiking Autonomous Driving (SAD), the first unified Spiking Neural Network (SNN) to address the energy challenges faced by autonomous driving systems through its event-driven and energy-efficient nature. SAD is trained end-to-end and consists of three main modules: perception, which processes inputs from multi-view cameras to construct a spatiotemporal bird's eye view; prediction, which utilizes a novel dual-pathway with spiking neurons to forecast future states; and planning, which generates safe trajectories considering predicted occupancy, traffic rules, and ride comfort. Evaluated on the nuScenes dataset, SAD achieves competitive performance in perception, prediction, and planning tasks, while drawing upon the energy efficiency of SNNs. This work highlights the potential of neuromorphic computing to be applied to energy-efficient autonomous driving, a critical step toward sustainable and safety-critical automotive technology. Our code is available at \url{https://github.com/ridgerchu/SAD}.

Figures (7)

• Figure 1: How SAD enables autonomous driving from vision to planning: The system processes inputs from six cameras across multiple frames. The perception module encodes feature information related to the present input frame ($T=n$), the prediction module predicts feature information of the next frame using sequential information ($T=n+1$), and the model output generates a steering and acceleration plan. This process creates a bird's eye view (BEV) and trajectory plan for navigation.
• Figure 2: Overview of SAD. The multi-view features from the perception encoder, including a spiking ResNet with inverted bottleneck and spiking DeepLab head, are fed into a prediction module using spiking neurons. The perception decoder then generates lane divider, pedestrian, vehicle and drivable area predictions. Finally, the planning module models the scene and generates future predictions to inform rule-based command decisions for turning, stopping, and goal-directed navigation.
• Figure 3: The perception module. The encoder takes multi-camera input data, passes it through a spiking ResNet with inverted bottleneck to generate feature representations, each of which has its own depth estimation. These are fused and passed to the decoder, which generates predictions for lane dividers, pedestrians, vehicles and drivable areas.
• Figure 4: Dual pathway modeling for prediction. Neuron $a$ captures future multi-modality by incorporating uncertainty distribution. Neuron $b$ compensates for information gaps using past variations. Inputs $x_1$ and $x_t$ from both pathways are used for the next prediction step.
• Figure 5: Qualitative Results of the SAD Model on the nuScenes Dataset. (a) displays six camera view inputs utilized by the model. (b) illustrates the planning result of the ANN model, and (c) presents the planning results of our SAD model. The comparison shows that our SAD model can achieve performance comparable to that of the ANN model and successfully generate a safe trajectory.