Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deep Reinforcement Learning with Spiking Q-learning

Ding Chen, Peixi Peng, Tiejun Huang, Yonghong Tian

TL;DR

DSQN presents end-to-end spike-based Q-learning by representing $Q(s,a)$ with the maximum membrane voltage of non-spiking neurons, enabling training and execution entirely within neuromorphic hardware. It builds on a standard Deep Q-learning objective, using backpropagation through time with a surrogate gradient for a spiking network, and demonstrates competitive or superior performance on 17 Atari games while improving robustness to adversarial attacks. Theoretical energy analysis indicates substantial energy savings over conventional ANN-based approaches, highlighting practical benefits for energy-constrained autonomous control. Overall, DSQN establishes a feasible path for fully spike-based reinforcement learning on neuromorphic platforms.

Abstract

With the help of special neuromorphic hardware, spiking neural networks (SNNs) are expected to realize artificial intelligence (AI) with less energy consumption. It provides a promising energy-efficient way for realistic control tasks by combining SNNs with deep reinforcement learning (RL). There are only a few existing SNN-based RL methods at present. Most of them either lack generalization ability or employ Artificial Neural Networks (ANNs) to estimate value function in training. The former needs to tune numerous hyper-parameters for each scenario, and the latter limits the application of different types of RL algorithm and ignores the large energy consumption in training. To develop a robust spike-based RL method, we draw inspiration from non-spiking interneurons found in insects and propose the deep spiking Q-network (DSQN), using the membrane voltage of non-spiking neurons as the representation of Q-value, which can directly learn robust policies from high-dimensional sensory inputs using end-to-end RL. Experiments conducted on 17 Atari games demonstrate the DSQN is effective and even outperforms the ANN-based deep Q-network (DQN) in most games. Moreover, the experiments show superior learning stability and robustness to adversarial attacks of DSQN.

Deep Reinforcement Learning with Spiking Q-learning

TL;DR

DSQN presents end-to-end spike-based Q-learning by representing with the maximum membrane voltage of non-spiking neurons, enabling training and execution entirely within neuromorphic hardware. It builds on a standard Deep Q-learning objective, using backpropagation through time with a surrogate gradient for a spiking network, and demonstrates competitive or superior performance on 17 Atari games while improving robustness to adversarial attacks. Theoretical energy analysis indicates substantial energy savings over conventional ANN-based approaches, highlighting practical benefits for energy-constrained autonomous control. Overall, DSQN establishes a feasible path for fully spike-based reinforcement learning on neuromorphic platforms.

Abstract

With the help of special neuromorphic hardware, spiking neural networks (SNNs) are expected to realize artificial intelligence (AI) with less energy consumption. It provides a promising energy-efficient way for realistic control tasks by combining SNNs with deep reinforcement learning (RL). There are only a few existing SNN-based RL methods at present. Most of them either lack generalization ability or employ Artificial Neural Networks (ANNs) to estimate value function in training. The former needs to tune numerous hyper-parameters for each scenario, and the latter limits the application of different types of RL algorithm and ignores the large energy consumption in training. To develop a robust spike-based RL method, we draw inspiration from non-spiking interneurons found in insects and propose the deep spiking Q-network (DSQN), using the membrane voltage of non-spiking neurons as the representation of Q-value, which can directly learn robust policies from high-dimensional sensory inputs using end-to-end RL. Experiments conducted on 17 Atari games demonstrate the DSQN is effective and even outperforms the ANN-based deep Q-network (DQN) in most games. Moreover, the experiments show superior learning stability and robustness to adversarial attacks of DSQN.
Paper Structure (26 sections, 20 equations, 7 figures, 6 tables)

This paper contains 26 sections, 20 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reward-based Learning by Three-factor Learning Rules
  4. ANN to SNN Conversion for RL
  5. Co-learning of Hybrid Framework
  6. RL methods using Spike-based BP
  7. Method
  8. Deep Q-learning
  9. Spiking Neural Model
  10. Non-spiking Neural Model
  11. Neural Coding
  12. Deep Spiking Q-network
  13. Training Framework
  14. Experiments
  15. Experimental Settings
  16. ...and 11 more sections

Figures (7)

  • Figure 1: The correspondence diagram between our method and the sensory motor neuron pathway.
  • Figure 2: The general neural model (Left) Spiking neural model. (Right) Non-spiking neural model.
  • Figure 3: The architecture of the simple network for the case study on different statistics.
  • Figure 4: The functional relationship between input current and statistics of membrane voltage.
  • Figure 5: The architecture of deep spiking Q-network.
  • ...and 2 more figures