Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-compartment Neuron and Population Encoding Powered Spiking Neural Network for Deep Distributional Reinforcement Learning

Yinqian Sun, Feifei Zhao, Zhuoya Zhao, Yi Zeng

TL;DR

This work tackles the challenge of enabling energy-efficient, biologically plausible deep distributional reinforcement learning with spiking neural networks. It introduces a brain-inspired framework, MCS-FQF, that combines a multi-compartment neuron (MCN) with population encoding to integrate state and quantile fraction information directly within a spiking architecture. Through end-to-end training via spatio-temporal backpropagation, MCS-FQF outperforms vanilla FQF and ANN-SNN conversion-based Spiking-FQF on 19 Atari games, with ablations showing the MCN and population encoding providing substantial performance and energy benefits. The study demonstrates the viability and practical impact of deep SNNs for complex decision-making tasks, highlighting potential gains in energy efficiency for neuromorphic RL deployments.

Abstract

Inspired by the brain's information processing using binary spikes, spiking neural networks (SNNs) offer significant reductions in energy consumption and are more adept at incorporating multi-scale biological characteristics. In SNNs, spiking neurons serve as the fundamental information processing units. However, in most models, these neurons are typically simplified, focusing primarily on the leaky integrate-and-fire (LIF) point neuron model while neglecting the structural properties of biological neurons. This simplification hampers the computational and learning capabilities of SNNs. In this paper, we propose a brain-inspired deep distributional reinforcement learning algorithm based on SNNs, which integrates a bio-inspired multi-compartment neuron (MCN) model with a population coding approach. The proposed MCN model simulates the structure and function of apical dendritic, basal dendritic, and somatic compartments, achieving computational power comparable to that of biological neurons. Additionally, we introduce an implicit fractional embedding method based on population coding of spiking neurons. We evaluated our model on Atari games, and the experimental results demonstrate that it surpasses the vanilla FQF model, which utilizes traditional artificial neural networks (ANNs), as well as the Spiking-FQF models that are based on ANN-to-SNN conversion methods. Ablation studies further reveal that the proposed multi-compartment neuron model and the quantile fraction implicit population spike representation significantly enhance the performance of MCS-FQF while also reducing power consumption.

Multi-compartment Neuron and Population Encoding Powered Spiking Neural Network for Deep Distributional Reinforcement Learning

TL;DR

This work tackles the challenge of enabling energy-efficient, biologically plausible deep distributional reinforcement learning with spiking neural networks. It introduces a brain-inspired framework, MCS-FQF, that combines a multi-compartment neuron (MCN) with population encoding to integrate state and quantile fraction information directly within a spiking architecture. Through end-to-end training via spatio-temporal backpropagation, MCS-FQF outperforms vanilla FQF and ANN-SNN conversion-based Spiking-FQF on 19 Atari games, with ablations showing the MCN and population encoding providing substantial performance and energy benefits. The study demonstrates the viability and practical impact of deep SNNs for complex decision-making tasks, highlighting potential gains in energy efficiency for neuromorphic RL deployments.

Abstract

Inspired by the brain's information processing using binary spikes, spiking neural networks (SNNs) offer significant reductions in energy consumption and are more adept at incorporating multi-scale biological characteristics. In SNNs, spiking neurons serve as the fundamental information processing units. However, in most models, these neurons are typically simplified, focusing primarily on the leaky integrate-and-fire (LIF) point neuron model while neglecting the structural properties of biological neurons. This simplification hampers the computational and learning capabilities of SNNs. In this paper, we propose a brain-inspired deep distributional reinforcement learning algorithm based on SNNs, which integrates a bio-inspired multi-compartment neuron (MCN) model with a population coding approach. The proposed MCN model simulates the structure and function of apical dendritic, basal dendritic, and somatic compartments, achieving computational power comparable to that of biological neurons. Additionally, we introduce an implicit fractional embedding method based on population coding of spiking neurons. We evaluated our model on Atari games, and the experimental results demonstrate that it surpasses the vanilla FQF model, which utilizes traditional artificial neural networks (ANNs), as well as the Spiking-FQF models that are based on ANN-to-SNN conversion methods. Ablation studies further reveal that the proposed multi-compartment neuron model and the quantile fraction implicit population spike representation significantly enhance the performance of MCS-FQF while also reducing power consumption.
Paper Structure (17 sections, 1 theorem, 40 equations, 12 figures, 7 tables, 1 algorithm)

This paper contains 17 sections, 1 theorem, 40 equations, 12 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Methods
  4. Multi-compartment Neuron Model for Information Integration
  5. Population Encoding For Quantile Fraction Implicit Representation
  6. Multi-compartment Spiking-FQF Model
  7. Experiments
  8. Performance of directly training multi-compartment spiking neural network
  9. Comparing with other SNN-based DRL models
  10. Ablation study on the effect of MCN and population encoding
  11. Computational complexity and energy efficiency
  12. Discussion
  13. Analysis about the spiking activity of MCN
  14. Analysis of MCN model in classification tasks
  15. Influence of hyperparameters on MCN models
  16. ...and 2 more sections

Key Result

Theorem 1

Let $Z=\frac{g_B+g_A+g_L}{\tau_L g_L}$, the somatic potential $u(t)$ is the spatial-temporal integration of the apical and basal dendritic potentials as:

Figures (12)

  • Figure 1: The overall architecture of MCS-FQF model.
  • Figure 2: Network architectures for FQF and MCS-FQF.
  • Figure 3: Point neuron model and multi-compartment neuron model. (A) Point neuron with soma compartment. (B) Computational model for point neuron with single information processing node. (C) Multi-compartment neuron with basal dendrite, apical dendrite and soma. (D) Computational model for multi-compartment neuron. (E) Structure and potentials influence of MCN. (F) Firing activity of MCN model. The potentials are recorded from constant inputs signal with $x_a=1.0$ and $x_b=1.5$ within $T=30$ time steps. The parameters of MCN model are set as $\tau_A=\tau_B=2.0$, $\tau_L=4.0$, $g_A=g_B=g_L=1.0$, and the threshold potential is $V_{th}=0.8$.
  • Figure 4: The spiking activity of population neuron. (A) Population neurons with Gaussian receptive field. The gray dotted line represents the relationship between the firing rate of different neurons and input stimuli. The intersection of the solid vertical line with the activity curve of each neuron is the firing rate of that neuron when encoding the corresponding quantile fraction value. (B) The spike numbers of $M=64$ neurons within $T=8$ time periods when representing different fraction values.
  • Figure 5: Encoding quantile fraction value with population neuron.
  • ...and 7 more figures

Theorems & Definitions (1)

  • Theorem 1