Detection of Fast-Moving Objects with Neuromorphic Hardware

TL;DR

This work evaluates the viability of deploying spiking neural networks on neuromorphic hardware for real-time robotic perception using event-based cameras. By benchmarking three SNN frameworks (sinabs, MetaTF, Lava) on three edge devices (DynapCNN, Akida, Loihi2) within a table-tennis robot scenario, the authors quantify ball-detection accuracy, latency, and power across offline and online experiments. They demonstrate that neuromorphic solutions can achieve sub-2-pixel localization accuracy with significantly lower energy than GPUs, while highlighting practical integration challenges (especially data transfer and PCIe connectivity). The study provides a publicly available event-based ball-detection dataset and a cross-device benchmark to guide future deployment and research in real-time neuromorphic robotic perception.

Abstract

Neuromorphic Computing (NC) and Spiking Neural Networks (SNNs) in particular are often viewed as the next generation of Neural Networks (NNs). NC is a novel bio-inspired paradigm for energy efficient neural computation, often relying on SNNs in which neurons communicate via spikes in a sparse, event-based manner. This communication via spikes can be exploited by neuromorphic hardware implementations very effectively and results in a drastic reductions of power consumption and latency in contrast to regular GPU-based NNs. In recent years, neuromorphic hardware has become more accessible, and the support of learning frameworks has improved. However, available hardware is partially still experimental, and it is not transparent what these solutions are effectively capable of, how they integrate into real-world robotics applications, and how they realistically benefit energy efficiency and latency. In this work, we provide the robotics research community with an overview of what is possible with SNNs on neuromorphic hardware focusing on real-time processing. We introduce a benchmark of three popular neuromorphic hardware devices for the task of event-based object detection. Moreover, we show that an SNN on a neuromorphic hardware is able to run in a challenging table tennis robot setup in real-time.

Figures (4)

• Figure 1: Left: Three examples of 2D ball detections in an accumulated event frame which serves as the input to the SNN with ground truth in green and the estimated position in red. Right: Five observed 2D trajectories in the camera frame of the event-based camera with ground truth in green and the estimated positions in red. Background: The table tennis robot setup with the robot hitting back a table tennis ball in a rally.
• Figure 2: Our camera setup consisting of four frame-based cameras (in blue) and two event-based cameras (in red) with baselines of $3$m to $5$m. Schematic is up to scale.
• Figure 3: The network has $128$ output neurons, which are split into two populations. Each neuron in the first population represents an $x$-position, and each neuron in the second represents a $y$-position. The target is also split into an $x$- and $y$-target, each setting one as the target activity for the correct neuron, $0.5$ for the two adjacent neurons, and zero for all others. Values are represented using brightness, with larger values being brighter.
• Figure 4: Left: Experiment setup for the offline experiment. Right: Experiment setup for the online experiment.