A Neuromorphic Approach to Obstacle Avoidance in Robot Manipulation

TL;DR

The paper addresses obstacle avoidance in robot manipulation by integrating event-based sensing with spiking neural networks to produce real-time trajectory adaptations. It presents a modular neuromorphic pipeline that processes event data through a convolutional SNN, decodes activations via first-spike-time coding, applies a potential-field-based avoidance directive, and updates a dynamic motion primitive for online motion planning, with PID control for execution. Validation includes both simulated and real Kinova Gen3 experiments, showing reliable collision avoidance where a non-adaptive baseline fails, robustness to different event-emulation schemes, and successful transfer to real EC data. The work highlights the potential of neuromorphic methods for energy-efficient, low-latency manipulation tasks and points to future avenues in SNN learning and neuromorphic hardware integration.

Abstract

Neuromorphic computing mimics computational principles of the brain in $\textit{silico}$ and motivates research into event-based vision and spiking neural networks (SNNs). Event cameras (ECs) exclusively capture local intensity changes and offer superior power consumption, response latencies, and dynamic ranges. SNNs replicate biological neuronal dynamics and have demonstrated potential as alternatives to conventional artificial neural networks (ANNs), such as in reducing energy expenditure and inference time in visual classification. Nevertheless, these novel paradigms remain scarcely explored outside the domain of aerial robots. To investigate the utility of brain-inspired sensing and data processing, we developed a neuromorphic approach to obstacle avoidance on a camera-equipped manipulator. Our approach adapts high-level trajectory plans with reactive maneuvers by processing emulated event data in a convolutional SNN, decoding neural activations into avoidance motions, and adjusting plans using a dynamic motion primitive. We conducted experiments with a Kinova Gen3 arm performing simple reaching tasks that involve obstacles in sets of distinct task scenarios and in comparison to a non-adaptive baseline. Our neuromorphic approach facilitated reliable avoidance of imminent collisions in simulated and real-world experiments, where the baseline consistently failed. Trajectory adaptations had low impacts on safety and predictability criteria. Among the notable SNN properties were the correlation of computations with the magnitude of perceived motions and a robustness to different event emulation methods. Tests with a DAVIS346 EC showed similar performance, validating our experimental event emulation. Our results motivate incorporating SNN learning, utilizing neuromorphic processors, and further exploring the potential of neuromorphic methods.

Figures (39)

• Figure 1: Images captured with a DAVIS346 EC. \ref{['dvs_images_intro:fast_object_rgb']} shows an RGB image of a fast-moving object and \ref{['dvs_images_intro:fast_object_events']} visualizes the captured events \ref{['dvs_images_intro:ego_motion_hybrid_1']} and \ref{['dvs_images_intro:ego_motion_hybrid_2']} show events superimposed on images captured while moving the camera.
• Figure 2: Neuronal dynamics in conventional ANNs (left) and SNNs (right). Gray/black borders signify inactivity/activity; only the bottom SNN neuron is active here, since it had just spiked.
• Figure 3: An illustration of spiking neuron dynamics.
• Figure 4: The main components and processing stages of our neuromorphic pipeline. The plot on the right shows a pre-planned trajectory (grey) and a resultant trajectory (green) adapted to avoid an obstacle (blue).
• Figure 5: Emulated events due to object motion on the right. Here, any event (ON or OFF) is indicated by a blue pixel on the event image. This was captured as the camera moved forward, inducing some events on the edges of distant objects.