Real-time processing of analog signals on accelerated neuromorphic hardware

TL;DR

This paper tackles efficient near-sensor sensory processing by bypassing conventional ADC/DAC stages and directly injecting continuous sensor signals into the analog compute cores of an accelerated neuromorphic platform. It demonstrates a fully on-chip processing pipeline on BrainScaleS-2, combining analog input, a Jeffress-inspired spiking network with delay chains, and embedded processor-driven actuator control to localize sound and align a servo-driven actuator in real time. Key contributions include the first continuous-valued sensor data injection into the analog neurons of the BrainScaleS-2 ASIC, and an end-to-end pipeline that processes input to action on-chip, achieving sub-millisecond latency and microsecond-scale delay dynamics that enable precise spatial localization. This approach enables rapid, low-power near-sensor processing and opens avenues for more complex, temporally precise neuromorphic systems and applications in fast, real-time sensing and actuation.

Abstract

Sensory processing with neuromorphic systems is typically done by using either event-based sensors or translating input signals to spikes before presenting them to the neuromorphic processor. Here, we offer an alternative approach: direct analog signal injection eliminates superfluous and power-intensive analog-to-digital and digital-to-analog conversions, making it particularly suitable for efficient near-sensor processing. We demonstrate this by using the accelerated BrainScaleS-2 mixed-signal neuromorphic research platform and interfacing it directly to microphones and a servo-motor-driven actuator. Utilizing BrainScaleS-2's 1000-fold acceleration factor, we employ a spiking neural network to transform interaural time differences into a spatial code and thereby predict the location of sound sources. Our primary contributions are the first demonstrations of direct, continuous-valued sensor data injection into the analog compute units of the BrainScaleS-2 ASIC, and actuator control using its embedded microprocessors. This enables a fully on-chip processing pipeline$\unicode{x2014}$from sensory input handling, via spiking neural network processing to physical action. We showcase this by programming the system to localize and align a servo motor with the spatial direction of transient noise peaks in real-time.

Figures (5)

• Figure 1: Photograph of the presented demonstrator. The is visible inside the black box on the left; it connects to a small with two microphones (silver capsules), and preprocessing circuits on the right. The owl figure turns, driven by a servo motor, toward the direction of detected sound sources.
• Figure 2: Schematic illustration of the Jeffress model for sound localization. The difference in distance of both ears to a sound source ($d_\text{L}-d_\text{R}$) induces a difference in the sound's arrival time, the itd. From the ears, the sound signal is transported to coincidence neurons (black) via nerve fibers. Depending on the distance traveled on the nerve fibers ($x_\text{L/R}$), delays of different lengths are introduced. Only where the respective accumulated delays compensate the itd, the coincidence neuron becomes active.
• Figure 3: Schematic overview over a single , adapted from pehle2022. The fully programmable synapse array allows the realization of near-arbitrary user-defined network topologies on the chip. The signal path used for injecting external, continuous-valued sensor data into the analog neuron circuits is highlighted in red.
• Figure 4: Sketch of the full signal processing pipeline. Sound signals are either captured via two microphones or mimicked by the sound card of a host computer (left). The analog signals, after passive conditioning to protect the chip circuitry (inset), directly stimulate the neuron circuits. The on-chip network implements a simplified version of the Jeffress model where the nerve-fiber delays are implemented as chains of connected neurons. The spike counts of the coincidence neurons are read out by the embedded processor, which produces an output signal to either operate the actuator, or to be parsed by the host computer (right).
• Figure 5: (A) Stereo recording of a person clapping, using the microphones depicted in \ref{['fig:system']}. One channel can be time delayed (orange) or advanced (blue) to mimic different sound source positions. (B) Rasterplots of the left (gray) and right (blue) delay chains (bottom) and coincidence detection neurons (top) for three different time delays between the left and right channel. (C) Variation over the sound direction as detected by the system for eight different itd, each over 100.0 trials. (D) Detected sound direction over itd between the input channels. For technical reasons, panel (B) has been recorded with artificially injected spikes to the respective first neuron. Panels (C) and (D) were recorded by playing back an audio recording of recorded claps (cf. \ref{['subsec:sensory-input']}).