Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GRSN: Gated Recurrent Spiking Neurons for POMDPs and MARL

Lang Qin, Ziming Wang, Runhao Jiang, Rui Yan, Huajin Tang

TL;DR

This work tackles the temporal mismatch in spiking reinforcement learning by introducing the Temporal Alignment Paradigm (TAP), which maps each spiking update to a single RL transition, enabling history accumulation within a single simulated window. To bolster memory capacity and temporal correlation in spiking neurons, the authors propose Gated Recurrent Spiking Neurons (GRSN) that incorporate forget and input gates via recurrent connections on the input current. The combination of TAP and GRSN achieves performance comparable to RNN-based RL on partially observable tasks and MARL benchmarks (SMAC), while substantially reducing energy consumption—up to 50% overall and up to orders of magnitude compared to RNNs in some cases. This approach demonstrates the viability of energy-efficient SNNs for complex sequential decision-making under partial observability and multi-agent coordination, with practical implications for neuromorphic deployments.

Abstract

Spiking neural networks (SNNs) are widely applied in various fields due to their energy-efficient and fast-inference capabilities. Applying SNNs to reinforcement learning (RL) can significantly reduce the computational resource requirements for agents and improve the algorithm's performance under resource-constrained conditions. However, in current spiking reinforcement learning (SRL) algorithms, the simulation results of multiple time steps can only correspond to a single-step decision in RL. This is quite different from the real temporal dynamics in the brain and also fails to fully exploit the capacity of SNNs to process temporal data. In order to address this temporal mismatch issue and further take advantage of the inherent temporal dynamics of spiking neurons, we propose a novel temporal alignment paradigm (TAP) that leverages the single-step update of spiking neurons to accumulate historical state information in RL and introduces gated units to enhance the memory capacity of spiking neurons. Experimental results show that our method can solve partially observable Markov decision processes (POMDPs) and multi-agent cooperation problems with similar performance as recurrent neural networks (RNNs) but with about 50% power consumption.

GRSN: Gated Recurrent Spiking Neurons for POMDPs and MARL

TL;DR

This work tackles the temporal mismatch in spiking reinforcement learning by introducing the Temporal Alignment Paradigm (TAP), which maps each spiking update to a single RL transition, enabling history accumulation within a single simulated window. To bolster memory capacity and temporal correlation in spiking neurons, the authors propose Gated Recurrent Spiking Neurons (GRSN) that incorporate forget and input gates via recurrent connections on the input current. The combination of TAP and GRSN achieves performance comparable to RNN-based RL on partially observable tasks and MARL benchmarks (SMAC), while substantially reducing energy consumption—up to 50% overall and up to orders of magnitude compared to RNNs in some cases. This approach demonstrates the viability of energy-efficient SNNs for complex sequential decision-making under partial observability and multi-agent coordination, with practical implications for neuromorphic deployments.

Abstract

Spiking neural networks (SNNs) are widely applied in various fields due to their energy-efficient and fast-inference capabilities. Applying SNNs to reinforcement learning (RL) can significantly reduce the computational resource requirements for agents and improve the algorithm's performance under resource-constrained conditions. However, in current spiking reinforcement learning (SRL) algorithms, the simulation results of multiple time steps can only correspond to a single-step decision in RL. This is quite different from the real temporal dynamics in the brain and also fails to fully exploit the capacity of SNNs to process temporal data. In order to address this temporal mismatch issue and further take advantage of the inherent temporal dynamics of spiking neurons, we propose a novel temporal alignment paradigm (TAP) that leverages the single-step update of spiking neurons to accumulate historical state information in RL and introduces gated units to enhance the memory capacity of spiking neurons. Experimental results show that our method can solve partially observable Markov decision processes (POMDPs) and multi-agent cooperation problems with similar performance as recurrent neural networks (RNNs) but with about 50% power consumption.
Paper Structure (34 sections, 1 theorem, 38 equations, 7 figures, 9 tables, 1 algorithm)

This paper contains 34 sections, 1 theorem, 38 equations, 7 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Spiking Reinforcement Learning
  4. Recurrent Spiking Neural Networks
  5. Preliminaries
  6. Notations
  7. Leaky Integrate-and-Fire Model
  8. Surrogate Gradient
  9. Method
  10. Temporal Alignment Paradigm
  11. Gradient Analysis of Spiking Neurons
  12. Gated Recurrent Spiking Neurons
  13. Experiments
  14. Partially Observable Classic Control Tasks
  15. Enviroments & Experimental Settings
  16. ...and 19 more sections

Key Result

Theorem 1

With the LIF model (Eq eq::Update-eq::Heaviside) and the common parameter settings ($\beta=0.5, \vartheta=1, \alpha=2, u_r=0$), the gradient of loss function $\frac{\partial \mathcal{L}}{\partial \bm{W}^l}$ mainly depends on current $\frac{\partial \bm{c}}{\partial \bm{W}^l}$ when total time-step $T

Figures (7)

  • Figure 1: Temporal mismatch issue in SRL algorithms. The figure shows the basic model of SRL methods; it uses spike results of multiple time steps to decode for one-step value function calculation or action selection in RL.
  • Figure 2: Overview of the proposed TAP and GRSN. A. Basic model of RNN-RL and SNN-RL algorithms under temporal alignment paradigm. Spike cell is a special form of single-step update for spiking neurons, which maintains the update of membrane potential $\bm{u}_t$ and outputs the $Q$ function. The multilayer perceptron (MLP) serves as the embedding or coding layer here. B. Detailed update dynamics for the parameters of spiking neurons. The state $s_t$ is encoded into the input current $\bm{c}_t$ and feedforward to the spiking neurons. The neurons calculate the current membrane potential $\bm{u}_t$ based on $\bm{c}_t$ and $\bm{u}_{t-1}$, and determine whether to emit spikes $\bm{o}_t$. The output spikes are decoded and used to obtain the action $a_t$, and then the next state $s_{t+1}$ is achieved based on the state transition function. C. Dynamics of GRSN. The red part shows the added recurrent connection and gated unit. $\mathscr{F}$ and $\mathscr{I}$ represent the forget gates and input gates, respectively. GRSN also follows the temporal alignment paradigm, with each time step corresponding to a single-step state transition.
  • Figure 3: Distributions of membrane potential $\bm{u}_t$ and the term $\beta \delta_t$. The y-axis represents the time step, and the x-axis represents the value of the parameter. The probability density curve corresponding to each time step represents the parameter distribution of all neurons in the single-layer network.
  • Figure 4: A. Diagram of the pendulum and cartpole problem. The agent needs to apply appropriate forces to control the system to reach the target state, in order to get rewards. B. Game scenarios of two different SMAC maps. Agents need to defeat enemies to achieve high rewards and win the game.
  • Figure 5: Results of classic partial observable control tasks. The shaded region represents a standard deviation of average evaluation over five different seeds, and the learning curves are smoothed for visual clarity.
  • ...and 2 more figures

Theorems & Definitions (2)

  • Theorem 1
  • proof