Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks

TL;DR

This work tackles two core challenges in training Spiking Neural Networks (SNNs) for reinforcement learning: non-differentiable spiking and the need for sequence-based training with warm-up periods. It analyzes surrogate-gradient slope settings and introduces an adaptive slope scheduling framework, showing that shallower slopes boost gradient magnitude in deeper layers but reduce gradient alignment, with RL particularly benefiting from these effects. The authors then propose TD3BC+JSRL, a Jump-Start Reinforcement Learning approach that uses a privileged guiding policy to bootstrap training and an asymmetric actor-critic setup to stabilize learning, achieving a final average reward of around 400 on a real quadrotor task and outperforming Behavioral Cloning and TD3 variants by a large margin. The combination of adaptive surrogate-gradient scheduling, privileged demonstrations, and online adaptation enables effective sequence-based learning in neuromorphic controllers, bridging the gap between simulation and real-world robotics while offering insights into energy-efficient neuromorphic control. Collectively, the paper advances both theoretical understanding of surrogate gradients in deep SNNs and practical training strategies for real-world neuromorphic RL systems.

Abstract

Neuromorphic computing systems are set to revolutionize energy-constrained robotics by achieving orders-of-magnitude efficiency gains, while enabling native temporal processing. Spiking Neural Networks (SNNs) represent a promising algorithmic approach for these systems, yet their application to complex control tasks faces two critical challenges: (1) the non-differentiable nature of spiking neurons necessitates surrogate gradients with unclear optimization properties, and (2) the stateful dynamics of SNNs require training on sequences, which in reinforcement learning (RL) is hindered by limited sequence lengths during early training, preventing the network from bridging its warm-up period. We address these challenges by systematically analyzing surrogate gradient slope settings, showing that shallower slopes increase gradient magnitude in deeper layers but reduce alignment with true gradients. In supervised learning, we find no clear preference for fixed or scheduled slopes. The effect is much more pronounced in RL settings, where shallower slopes or scheduled slopes lead to a 2.1x improvement in both training and final deployed performance. Next, we propose a novel training approach that leverages a privileged guiding policy to bootstrap the learning process, while still exploiting online environment interactions with the spiking policy. Combining our method with an adaptive slope schedule for a real-world drone position control task, we achieve an average return of 400 points, substantially outperforming prior techniques, including Behavioral Cloning and TD3BC, which achieve at most --200 points under the same conditions. This work advances both the theoretical understanding of surrogate gradient learning in SNNs and practical training methodologies for neuromorphic controllers demonstrated in real-world robotic systems.

Figures (5)

• Figure 1: Surrogate gradient slope-magnitude-alignment tradeoff in deep SNNs. (\ref{['fig:surrslope']}) Shallow slopes ($k{=}1$) provide gradients over wider input range than steep slopes ($k{=}100$). (\ref{['fig:gradmagn']}) This prevents vanishing gradients in deeper layers, critical for multi-layer SNN training. (\ref{['fig:cossim']}) However, shallow slopes introduce directional noise (cosine similarity $\to$ 0), which aids RL exploration but hinders supervised convergence. Network: 4 layers of 64 neurons; layer 0 = first hidden, layer 4 = output.
• Figure 2: The spiking controller receives $(x, y, z)$, linear velocity $(v_x, v_y, v_z)$, orientation angles $(\theta, \phi, \psi)$, and angular velocities $(p, q, r)$ inputs and outputs motor commands $(m_1, m_2, m_3, m_4)$.
• Figure 3: Comparison between fixed and scheduled surrogate gradient slopes for both BC and RL settings. While BC performance is largely slope-invariant, RL benefits from shallower or scheduled slopes, which improve exploration and training efficiency. Scheduling effectively balances stability and performance, reducing the need for extensive slope tuning.
• Figure 4: Comparison of training approaches under a reward curriculum. All experiments were run with 5 different seeds. As the curriculum increases difficulty of the task, we find that only TD3BC+JSRL is able to adapt to the changing curriculum. During testing, we average the results from 20 runs.
• Figure 5: When deployed on the Crazyflie, the spiking actor exhibits oscillatory behavior but always successfully performs maneuvers such as circular flight.