S2Act: Simple Spiking Actor

Abstract

Spiking neural networks (SNNs) and biologically-inspired learning mechanisms are attractive in mobile robotics, where the size and performance of onboard neural network policies are constrained by power and computational budgets. Existing SNN approaches, such as population coding, reward modulation, and hybrid artificial neural network (ANN)-SNN architectures, have shown promising results; however, they face challenges in complex, highly stochastic environments due to SNN sensitivity to hyperparameters and inconsistent gradient signals. To address these challenges, we propose simple spiking actor (S2Act), a computationally lightweight framework that deploys an RL policy using an SNN in three steps: (1) architect an actor-critic model based on an approximated network of rate-based spiking neurons, (2) train the network with gradients using compatible activation functions, and (3) transfer the trained weights into physical parameters of rate-based leaky integrate-and-fire (LIF) neurons for inference and deployment. By globally shaping LIF neuron parameters such that their rate-based responses approximate ReLU activations, S2Act effectively mitigates the vanishing gradient problem, while pre-constraining LIF response curves reduces reliance on complex SNN-specific hyperparameter tuning. We demonstrate our method in two multi-agent stochastic environments (capture-the-flag and parking) that capture the complexity of multi-robot interactions, and deploy our trained policies on physical TurtleBot platforms using Intel's Loihi neuromorphic hardware. Our experimental results show that S2Act outperforms relevant baselines in task performance and real-time inference in nearly all considered scenarios, highlighting its potential for rapid prototyping and efficient real-world deployment of SNN-based RL policies.

Figures (6)

• Figure 1: S2Act. An ANN with rate-based LIF activation is used for training, with learned weights then converted to a spiking network for deployment on neuromorphic hardware.
• Figure 2: S2Act training-to-deployment pipeline. (a) We employ an ANN-to-SNN conversion strategy, enabling neuromorphic deployment of computationally lightweight SNN RL agents. In the rate-based LIF network training phase, the simulated environment provides discrete visual observations ${O_{t}}$. These are passed to an actor-critic model, comprising a policy (actor) network and a value (critic) network, both modeled with soft-ReLLIF units (green circles). We use PPO to train these networks. (b) Once trained, the soft-ReLLIF neurons in the actor network are replaced with spiking LIF neurons (blue circles) for deployment on neuromorphic hardware, such as Intel’s Loihi.
• Figure 3: Rate-based LIF neuron variants. Rate-based LIF units (cyan line) exhibit unbounded gradients at the critical input value determined by their neuronal parameters. Although a smoothing term can be used to address these unbounded gradients, the resulting soft rate-based LIF unit (purple line with star markers) will still be prone to vanishing gradients. A common solution to this problem in networks with bounded activation functions is to employ ReLU activation. Therefore, we adjust the neuronal parameters of the soft rate-based LIF units to approximate the ReLU activation. Consequently, we utilize soft ReLLIF (green line) activations.
• Figure 4: Simulated and real-world environment for evaluations. (a) Visualization of our simulated 2 vs. 2 CtF game. Gray squares are obstacles, triangles are agents, and circles are flags. The region highlighted by solid red lines is the border region for a red agent whose policy is the patrol policy. The game has stochastic combat: when two agents are adjacent, whether one is eliminated depends on the territory, the number of nearby enemies, and the number of nearby allies. (b) Real-world CtF arena measures 12' $\times$ 12' and replicates the 10 $\times$ 10 grid-world used during policy training. Blue, red, and gray tiles denote the blue team's territory, the red team's territory, and obstacles, respectively. (c) A Qualisys motion capture system with ten Miqus M5 cameras and Qualisys Track Manager software is used to track robot positions in real time. (d) Visualization of the parking environment. The green rectangle is the ego vehicle, and the blue square is the target parking spot.
• Figure 5: Evaluation results. Average training curves of S2Act and other baselines in two CtF scenarios (a, b) and the parking environment (c). Solid lines represent the average mean episodic reward over three seeds with shaded areas representing one standard deviation confidence intervals.