The More the Merrier: Running Multiple Neuromorphic Components On-Chip for Robotic Control

TL;DR

This work shows a first example of a pipeline for vision-based robot control in which numerous complex networks can be run entirely on hardware via the use of a spiking neural state machine for process orchestration.

Abstract

It has long been realized that neuromorphic hardware offers benefits for the domain of robotics such as low energy, low latency, as well as unique methods of learning. In aiming for more complex tasks, especially those incorporating multimodal data, one hurdle continuing to prevent their realization is an inability to orchestrate multiple networks on neuromorphic hardware without resorting to off-chip process management logic. To address this, we show a first example of a pipeline for vision-based robot control in which numerous complex networks can be run entirely on hardware via the use of a spiking neural state machine for process orchestration. The pipeline is validated on the Intel Loihi 2 research chip. We show that all components can run concurrently on-chip in the milli Watt regime at latencies competitive with the state-of-the-art. An equivalent network on simulated hardware is shown to accomplish robotic arm plug insertion in simulation, and the core elements of the pipeline are additionally tested on a real robotic arm.

Figures (10)

• Figure 1: Pipeline flow diagram. A full architecture with the NSM components is included in Figure \ref{['fig:full_architecture']}.
• Figure 2: RN Architecture. Each square represents a 2D field of neurons --- larger squares are $80\times80$, smaller squares are $80\times40$. The input creates a region of activity in the selective DNF (left). These are then both connected to a "select neuron field" (center large square). Four weight matrices $M_{i,j}$ connect the quadrants to a single output LIF field (right), hence completing the gating. The Input and DNF contain LIF neurons, while the select neuron field and gated field are both LIF Reset.
• Figure 3: For visual servoing the DNF is connected to two $80\times80$ directional fields in which regions correspond to directions of robot motion (these are in fact abstractions of the weight matrix connecting the selective DNF to the 5 directional neurons). Within these fields the colors represent the following end-effector translations: blue = $+y$, yellow = $-y$, red = $-x$, green = $+x$, purple = $-z$. The shaded region indicates where the tip of the end-effector peg occludes a region of the FoV in the general case (in our case we adjust the camera such that the occlusion is negligible -- see section \ref{['sec:sim']}).
• Figure 4: Full pipeline architecture. The pipeline is divided into two elementary behaviors (EBs). The first comprises the search behavior and the second the visual servoing. Violet boxes represent the logic blocks guiding the NSM: I = Intention, C = Condition of Satisfaction, D = Condition of Dissatisfaction. Additionally, P is the Precondition node (which initiates the next EB).
• Figure 5: An illustration of when various components within the architecture are seen to spike. Note that for neuronal fields such as the DNF we consider the layer to spike if at least one neuron is spiking. The inlaid images represent the voltages of the neuron fields for given timesteps. These are generated using LAVA voltage probes when running the network on real Loihi 2. Summary of inlays from left to right: event input, initial DNF selection, RN gated field, first peak stored in memory DNF, selective DNF peak collapsed by CoD (see single spike at timestep 70), new peak forms while previous socket is inhibited by memory DNF, new peak location is stored in memory DNF. Color scale is set such that black = minimum voltage (varies between fields and timestepsa) and white $\geq v_{th}$ (spiking neurons). Note the EB2 CoS is designed to only spike when the plug makes contact with the socketboard. As here the input is pre-recorded and the output is not connected to the robot, this is not possible, and hence there is no spike.