Signal-Adaptive Trust Regions for Gradient-Free Optimization of Recurrent Spiking Neural Networks

TL;DR

Inspired by trust-region methods in reinforcement learning that constrain policy updates in distribution space, SATR is proposed, a distributional update rule that constrains relative change by bounding KL divergence normalized by an estimated signal energy.

Abstract

Recurrent spiking neural networks (RSNNs) are a promising substrate for energy-efficient control policies, but training them for high-dimensional, long-horizon reinforcement learning remains challenging. Population-based, gradient-free optimization circumvents backpropagation through non-differentiable spike dynamics by estimating gradients. However, with finite populations, high variance of these estimates can induce harmful and overly aggressive update steps. Inspired by trust-region methods in reinforcement learning that constrain policy updates in distribution space, we propose \textbf{Signal-Adaptive Trust Regions (SATR)}, a distributional update rule that constrains relative change by bounding KL divergence normalized by an estimated signal energy. SATR automatically expands the trust region under strong signals and contracts it when updates are noise-dominated. We instantiate SATR for Bernoulli connectivity distributions, which have shown strong empirical performance for RSNN optimization. Across a suite of high-dimensional continuous-control benchmarks, SATR improves stability under limited populations and reaches competitive returns against strong baselines including PPO-LSTM. In addition, to make SATR practical at scale, we introduce a bitset implementation for binary spiking and binary weights, substantially reducing wall-clock training time and enabling fast RSNN policy search.

Figures (6)

• Figure 1: Reward--runtime trade-off for RSNN policy optimization. Final episodic return versus end-to-end wall-clock training time on Humanoid, Hopper, and Walker2d. For population-based methods, each marker corresponds to one population size, and lines connect settings within the same method. Our approach ($\text{SATR-RSNN}^*$) achieves a more favorable reward--runtime trade-off, particularly in the limited-population regime.
• Figure 2: Performance Benchmarks Across Tasks and Population Scales. Learning curves for three continuous control tasks under varying population budgets ($N$). Each column represents a distinct population size, demonstrating the scalability of our approach. SATR (ours) consistently achieves the highest terminal returns and exhibits superior convergence stability compared to EC and ES.
• Figure S1: Performance comparison between PPO-LSTM and SATR-RSNN* (population = 4096) on the Humanoid task under matched computational budgets. Learning curves are smoothed for visualization.
• Figure S2: Performance comparison between PPO-LSTM and SATR-RSNN* (population = 8192) on the Humanoid task under matched computational budgets. Learning curves are smoothed for visualization.
• Figure S3: Performance comparison between PPO-LSTM and SATR-RSNN* without acceleration (population = 8192) on the Humanoid task under matched computational budgets. Learning curves are smoothed for visualization.