Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Advancing Spiking Neural Networks for Sequential Modeling with Central Pattern Generators

Changze Lv, Dongqi Han, Yansen Wang, Xiaoqing Zheng, Xuanjing Huang, Dongsheng Li

TL;DR

This work demonstrates that the commonly used sinusoidal PE is mathematically a specific solution to the membrane potential dynamics of a particular CPG, and proposes a novel PE technique for SNNs, termed CPG-PE, which outperform their conventional counterparts.

Abstract

Spiking neural networks (SNNs) represent a promising approach to developing artificial neural networks that are both energy-efficient and biologically plausible. However, applying SNNs to sequential tasks, such as text classification and time-series forecasting, has been hindered by the challenge of creating an effective and hardware-friendly spike-form positional encoding (PE) strategy. Drawing inspiration from the central pattern generators (CPGs) in the human brain, which produce rhythmic patterned outputs without requiring rhythmic inputs, we propose a novel PE technique for SNNs, termed CPG-PE. We demonstrate that the commonly used sinusoidal PE is mathematically a specific solution to the membrane potential dynamics of a particular CPG. Moreover, extensive experiments across various domains, including time-series forecasting, natural language processing, and image classification, show that SNNs with CPG-PE outperform their conventional counterparts. Additionally, we perform analysis experiments to elucidate the mechanism through which SNNs encode positional information and to explore the function of CPGs in the human brain. This investigation may offer valuable insights into the fundamental principles of neural computation. Our code is available at https://github.com/microsoft/SeqSNN.

Advancing Spiking Neural Networks for Sequential Modeling with Central Pattern Generators

TL;DR

This work demonstrates that the commonly used sinusoidal PE is mathematically a specific solution to the membrane potential dynamics of a particular CPG, and proposes a novel PE technique for SNNs, termed CPG-PE, which outperform their conventional counterparts.

Abstract

Spiking neural networks (SNNs) represent a promising approach to developing artificial neural networks that are both energy-efficient and biologically plausible. However, applying SNNs to sequential tasks, such as text classification and time-series forecasting, has been hindered by the challenge of creating an effective and hardware-friendly spike-form positional encoding (PE) strategy. Drawing inspiration from the central pattern generators (CPGs) in the human brain, which produce rhythmic patterned outputs without requiring rhythmic inputs, we propose a novel PE technique for SNNs, termed CPG-PE. We demonstrate that the commonly used sinusoidal PE is mathematically a specific solution to the membrane potential dynamics of a particular CPG. Moreover, extensive experiments across various domains, including time-series forecasting, natural language processing, and image classification, show that SNNs with CPG-PE outperform their conventional counterparts. Additionally, we perform analysis experiments to elucidate the mechanism through which SNNs encode positional information and to explore the function of CPGs in the human brain. This investigation may offer valuable insights into the fundamental principles of neural computation. Our code is available at https://github.com/microsoft/SeqSNN.
Paper Structure (43 sections, 2 theorems, 22 equations, 5 figures, 5 tables)

This paper contains 43 sections, 2 theorems, 22 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Spiking Neural Networks
  4. Positional Encoding
  5. Central Pattern Generators
  6. Methods
  7. Relationship between Central Pattern Generators and Positional Encoding
  8. CPG-based Positional Encoding
  9. Implementations
  10. Experiments
  11. Datasets
  12. Time-Series Forecasting
  13. Text Classification
  14. Image Classification
  15. Sweeping CPG properties
  16. ...and 28 more sections

Key Result

Lemma 1

Given an LIF neuron defined by equ:membranePotentialequ:htequ:s(t) with decay rate $\beta$ and threshold $U_{thr}$, starting with resting potential $U(0)=0$, if fed with the constant current $I(t)=I_c > 0$, the first spike will emit at:

Figures (5)

  • Figure 1: (a) Positional encoding (PE) in ANN Transformers. (b) Relative PE in Spike Transformers Zhou2022SpikformerWSyao2023spikeyao2024spikedriven. (c) Our Proposed CPG-PE method. (d) CPG-PE consistently improves learning performance across various tasks. CPG-PE is an ideal PE method tailored for SNNs, detailed in \ref{['sec:method']}.
  • Figure 2: (a) Illustration of a pair of CPG neurons demonstrating mutual inhibition through spiking activity. The spikes represent neural spikes that inhibit each other, exemplifying the coordination mechanism in CPG networks. (b) Spike trains of the first $4$ CPG neurons. The curve represents the membrane potential, while the vertical lines represent spikes.
  • Figure 3: Illustration of applying CPG-PE to SNNs. $X$, $X'$, and $X_{output}$ are all spike matrices.
  • Figure 4: (a)(c) $R^2$ versus $\tau$ and $N$ on time-series forecasting tasks. (b)(d) Accuracy versus $\tau$ and $N$ on image classification tasks. $\tau \in \{100,1000,5000,10000\}$, $N \in \{5,10,20\}$.
  • Figure 5: An illustration of the implementation of integrating a CPG-PE into a linear layer.

Theorems & Definitions (4)

  • Lemma 1
  • proof
  • Theorem 1
  • proof