Fully Asynchronous Neuromorphic Perception for Mobile Robot Dodging with Loihi Chips

TL;DR

This work presents the first fully asynchronous neuromorphic perception framework for a mobile robot dodging task by tightly integrating an event camera, spiking neural networks, and Intel Loihi hardware. A Key-Event-Point (KEP) module preserves essential temporal information while drastically reducing event counts, enabling efficient on-chip inference via a dodging network trained with simulated SNNs and deployed with 8-bit weight quantization. Empirical results show superior robustness to time-window and lighting variations compared to frame-based and other spike-based baselines, with orders-of-magnitude lower power than GPU-based or frame-structured approaches. The system demonstrates real-world feasibility on a fully neuromorphic mobile robot, offering a brain-inspired, low-power solution for dynamic obstacle avoidance in constrained platforms.

Abstract

Sparse and asynchronous sensing and processing in natural organisms lead to ultra low-latency and energy-efficient perception. Event cameras, known as neuromorphic vision sensors, are designed to mimic these characteristics. However, fully utilizing the sparse and asynchronous event stream remains challenging. Influenced by the mature algorithms of standard cameras, most existing event-based algorithms still rely on the "group of events" processing paradigm (e.g., event frames, 3D voxels) when handling event streams. This paradigm encounters issues such as feature loss, event stacking, and high computational burden, which deviates from the intended purpose of event cameras. To address these issues, we propose a fully asynchronous neuromorphic paradigm that integrates event cameras, spiking networks, and neuromorphic processors (Intel Loihi). This paradigm can faithfully process each event asynchronously as it arrives, mimicking the spike-driven signal processing in biological brains. We compare the proposed paradigm with the existing "group of events" processing paradigm in detail on the real mobile robot dodging task. Experimental results show that our scheme exhibits better robustness than frame-based methods with different time windows and light conditions. Additionally, the energy consumption per inference of our scheme on the embedded Loihi processor is only 4.30% of that of the event spike tensor method on NVIDIA Jetson Orin NX with energy-saving mode, and 1.64% of that of the event frame method on the same neuromorphic processor. As far as we know, this is the first time that a fully asynchronous neuromorphic paradigm has been implemented for solving sequential tasks on real mobile robot.

Figures (8)

• Figure 1: Comparison between synchronous and asynchronous paradigms for event streams. In the asynchronous paradigm, events are processed asynchronously according to their triggered time. The triggered neurons send spikes that are processed asynchronously. In the synchronous paradigm, event streams are converted to the image-like reprensentation and information is processed synchronously by every neuron.
• Figure 2: Offline training and online inference of dodging network. The offline simulated SNN is trained on GPU with a specific loss function and fast loss backpropagation. Then the pre-trained SNN is deployed to Kapoho Bay using weight integer quantization to achieve online inference. Finally, the output spike trains are decoded to dodging action.
• Figure 3: Event camera asynchronous processing. The events first are converted to address sequence, containing the chip, neurocore, and neuron address. And time flags separate addresses at different times. Finally, the event stream is sent to Loihi asynchronous with embedded lakemont (x86) processors.
• Figure 4: The fully neuromorphic mobile robot system consisting of a mecanum wheel chassis, a STM32 controller, a infrared transmitter, a host computer, an event camera (DAVIS 346), and a neuromorphic processor (Kapoho Bay). The host computer extracts the key event stream from the raw event stream and converts the event stream into the address sequence. Then the host computer sends events to corresponding neurons with embedded x86 processors under NxSDK. After Kapoho Bay completes the inference, the host computer decodes the dodging action based on the output spike trains.
• Figure 5: Some frame images, event frames of the raw and key event stream of datasets under outdoor normal light (10000 Lux), indoor normal light (1200 Lux), outdoor low light (25 Lux) and indoor low light (15 Lux). Frame images are only for visualization. Raw and key event streams are converted to event frames for visualization only. (a)-(c) are under outdoor normal light. (d)-(f) are under indoor normal light. (g)-(i) are under outdoor low light. (j)-(l) are under indoor low light.