Embedded event based object detection with spiking neural network

TL;DR

This work presents an embedded approach for event-based object detection using spiking neural networks on a low-power FPGA accelerator (SPLEAT) controlled by the Qualia framework. It introduces SpikeThin-VGG as a compact SNN backbone and investigates deployment strategies that partition computation between SPLEAT and CPU, enabling real-world OD with low energy cost. The study demonstrates feasible feasibility with Google Speech Commands as a preliminary test and shows promising energy efficiency and latency trade-offs for automotive OD on Gen1 event data, achieving a notable energy-per-output reduction and near-complete end-to-end deployment. The results indicate that, with planned parallelism and architectural enhancements, SPLEAT-based embedded SNNs can approach real-time, ultra-low-power operation suitable for near-sensor processing in autonomous systems.

Abstract

The complexity of event-based object detection (OD) poses considerable challenges. Spiking Neural Networks (SNNs) show promising results and pave the way for efficient event-based OD. Despite this success, the path to efficient SNNs on embedded devices remains a challenge. This is due to the size of the networks required to accomplish the task and the ability of devices to take advantage of SNNs benefits. Even when "edge" devices are considered, they typically use embedded GPUs that consume tens of watts. In response to these challenges, our research introduces an embedded neuromorphic testbench that utilizes the SPiking Low-power Event-based ArchiTecture (SPLEAT) accelerator. Using an extended version of the Qualia framework, we can train, evaluate, quantize, and deploy spiking neural networks on an FPGA implementation of SPLEAT. We used this testbench to load a state-of-the-art SNN solution, estimate the performance loss associated with deploying the network on dedicated hardware, and run real-world event-based OD on neuromorphic hardware specifically designed for low-power spiking neural networks. Remarkably, our embedded spiking solution, which includes a model with 1.08 million parameters, operates efficiently with 490 mJ per prediction.

Figures (3)

• Figure 1: ST-VGG+SSD architecture b29 with the layer distribution plan where c stands for channels, s stride and 3x3 kernel's size.
• Figure 2: Mean spike accumulation and activity per output per video clip for the 16 bit quantized Backbone network on workstation CPU with a total mean of 215076 spikes per video clip and 32.05% activity.
• Figure 3: Output bounding boxes of the small 32-ST-VGG with its backbone on $CPU_{float}$ (red), $CPU_{16 bits}$ (green), $SPLEAT$ (blue). $50$ms event frame extracted from the 1200 event-frames of the 60s video clip : https://leat.univ-cotedazur.fr/edge