Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Training slow silicon neurons to control extremely fast robots with spiking reinforcement learning

Irene Ambrosini, Ingo Blakowski, Dmitrii Zendrikov, Cristiano Capone, Luna Gava, Giacomo Indiveri, Chiara De Luca, Chiara Bartolozzi

TL;DR

Addresses the challenge of energy-efficient, real-time motor control in autonomous robotics by applying neuromorphic reinforcement learning to a 6D continuous-state air-hockey task ($x_p,y_p,v_x,v_y,x_e,y_e$). The approach couples a CPU-based encoder with a DYNAP-SE neuromorphic chip, a fixed random reservoir of 1020 AdEx-LIF neurons, and a two-unit readout learned online with the local e-prop rule, achieving 50 Hz closed-loop control. The authors demonstrate scaling from 2D Pong to this real robot, reaching 96–98% success across varied initial conditions with about 1020 neurons and online learning. The results highlight practical viability of energy-efficient, always-on adaptive robotics and motivate future hardware scaling, offline dream-like learning, and richer sensor interfaces.

Abstract

Air hockey demands split-second decisions at high puck velocities, a challenge we address with a compact network of spiking neurons running on a mixed-signal analog/digital neuromorphic processor. By co-designing hardware and learning algorithms, we train the system to achieve successful puck interactions through reinforcement learning in a remarkably small number of trials. The network leverages fixed random connectivity to capture the task's temporal structure and adopts a local e-prop learning rule in the readout layer to exploit event-driven activity for fast and efficient learning. The result is real-time learning with a setup comprising a computer and the neuromorphic chip in-the-loop, enabling practical training of spiking neural networks for robotic autonomous systems. This work bridges neuroscience-inspired hardware with real-world robotic control, showing that brain-inspired approaches can tackle fast-paced interaction tasks while supporting always-on learning in intelligent machines.

Training slow silicon neurons to control extremely fast robots with spiking reinforcement learning

TL;DR

Addresses the challenge of energy-efficient, real-time motor control in autonomous robotics by applying neuromorphic reinforcement learning to a 6D continuous-state air-hockey task (). The approach couples a CPU-based encoder with a DYNAP-SE neuromorphic chip, a fixed random reservoir of 1020 AdEx-LIF neurons, and a two-unit readout learned online with the local e-prop rule, achieving 50 Hz closed-loop control. The authors demonstrate scaling from 2D Pong to this real robot, reaching 96–98% success across varied initial conditions with about 1020 neurons and online learning. The results highlight practical viability of energy-efficient, always-on adaptive robotics and motivate future hardware scaling, offline dream-like learning, and richer sensor interfaces.

Abstract

Air hockey demands split-second decisions at high puck velocities, a challenge we address with a compact network of spiking neurons running on a mixed-signal analog/digital neuromorphic processor. By co-designing hardware and learning algorithms, we train the system to achieve successful puck interactions through reinforcement learning in a remarkably small number of trials. The network leverages fixed random connectivity to capture the task's temporal structure and adopts a local e-prop learning rule in the readout layer to exploit event-driven activity for fast and efficient learning. The result is real-time learning with a setup comprising a computer and the neuromorphic chip in-the-loop, enabling practical training of spiking neural networks for robotic autonomous systems. This work bridges neuroscience-inspired hardware with real-world robotic control, showing that brain-inspired approaches can tackle fast-paced interaction tasks while supporting always-on learning in intelligent machines.
Paper Structure (13 sections, 2 equations, 2 figures, 1 table)

This paper contains 13 sections, 2 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Our approach and contribution
  3. Methods
  4. Experimental Setup
  5. Network architecture
  6. Training reward
  7. Training and Evaluation Procedure
  8. Scaling from Pong to Physical Robotics
  9. Experimental conditions and results
  10. Task-level generalization
  11. Encoding-range scalability
  12. Behavior consolidation through learning
  13. Discussion

Figures (2)

  • Figure 1: Control pipeline and environment.Top-left: High-level flow from MuJoCo (puck $[x_p,y_p,v_x,v_y]$ and end-effector $[x_{ee},y_{ee}]$) through the decision module to the robot controller. The CPU encodes sensory data into spike trains, processed by DYNAP-SE' silicon neurons, then decoded into discrete motion primitives and translated to joint commands $[q_1,q_2,q_3]$. Bottom-left: Encoding module, reservoir, and spike-based readout with learning. Right: Table dimensions (1.038m $\times$ 1.948m), puck/mallet radii, puck generation region, and two arrival points for Action 0/1 in the robot frame.
  • Figure 2: Neuromorphic learning masters interception timing and generalizes robustly.(a) Timing acquisition: Pre-training (dashed) shows erratic actions; post-training (solid) achieves immediate, low-variance interceptions, reflecting learned timing. (b) Policy evolution: Stochastic switching between motion primitives consolidates into a deterministic sequence—early commitment followed by precise, tactical switching. (c) Generalization:Left: Learning curves for three initial velocity ranges. Right: Robustness panel comparing convergence across dynamic conditions.