Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Topological Representations of Heterogeneous Learning Dynamics of Recurrent Spiking Neural Networks

Biswadeep Chakraborty, Saibal Mukhopadhyay

TL;DR

A novel methodology to use RTD to measure the difference between distributed representations of RSNN models with different learning methods is introduced and heterogeneous STDP in RSNNs yield distinct representations than their homogeneous and surrogate gradient-based supervised learning counterparts.

Abstract

Spiking Neural Networks (SNNs) have become an essential paradigm in neuroscience and artificial intelligence, providing brain-inspired computation. Recent advances in literature have studied the network representations of deep neural networks. However, there has been little work that studies representations learned by SNNs, especially using unsupervised local learning methods like spike-timing dependent plasticity (STDP). Recent work by \cite{barannikov2021representation} has introduced a novel method to compare topological mappings of learned representations called Representation Topology Divergence (RTD). Though useful, this method is engineered particularly for feedforward deep neural networks and cannot be used for recurrent networks like Recurrent SNNs (RSNNs). This paper introduces a novel methodology to use RTD to measure the difference between distributed representations of RSNN models with different learning methods. We propose a novel reformulation of RSNNs using feedforward autoencoder networks with skip connections to help us compute the RTD for recurrent networks. Thus, we investigate the learning capabilities of RSNN trained using STDP and the role of heterogeneity in the synaptic dynamics in learning such representations. We demonstrate that heterogeneous STDP in RSNNs yield distinct representations than their homogeneous and surrogate gradient-based supervised learning counterparts. Our results provide insights into the potential of heterogeneous SNN models, aiding the development of more efficient and biologically plausible hybrid artificial intelligence systems.

Topological Representations of Heterogeneous Learning Dynamics of Recurrent Spiking Neural Networks

TL;DR

A novel methodology to use RTD to measure the difference between distributed representations of RSNN models with different learning methods is introduced and heterogeneous STDP in RSNNs yield distinct representations than their homogeneous and surrogate gradient-based supervised learning counterparts.

Abstract

Spiking Neural Networks (SNNs) have become an essential paradigm in neuroscience and artificial intelligence, providing brain-inspired computation. Recent advances in literature have studied the network representations of deep neural networks. However, there has been little work that studies representations learned by SNNs, especially using unsupervised local learning methods like spike-timing dependent plasticity (STDP). Recent work by \cite{barannikov2021representation} has introduced a novel method to compare topological mappings of learned representations called Representation Topology Divergence (RTD). Though useful, this method is engineered particularly for feedforward deep neural networks and cannot be used for recurrent networks like Recurrent SNNs (RSNNs). This paper introduces a novel methodology to use RTD to measure the difference between distributed representations of RSNN models with different learning methods. We propose a novel reformulation of RSNNs using feedforward autoencoder networks with skip connections to help us compute the RTD for recurrent networks. Thus, we investigate the learning capabilities of RSNN trained using STDP and the role of heterogeneity in the synaptic dynamics in learning such representations. We demonstrate that heterogeneous STDP in RSNNs yield distinct representations than their homogeneous and surrogate gradient-based supervised learning counterparts. Our results provide insights into the potential of heterogeneous SNN models, aiding the development of more efficient and biologically plausible hybrid artificial intelligence systems.
Paper Structure (13 sections, 3 equations, 4 figures, 2 tables)

This paper contains 13 sections, 3 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Related Works:
  4. Heterogeneous RSNN
  5. Similarity in Representations
  6. Methods
  7. General Framework
  8. Dual Representation for Recurrent Layers
  9. Comparing SNN Representations using RTD
  10. Experiments
  11. Datasets and Models
  12. Results
  13. Conclusions

Figures (4)

  • Figure 1: Figure showing the heterogeneity in the synaptic parameters
  • Figure 2: Figure showing how the layers are formulated in the recurrent layer
  • Figure 3: Layer 3, 5 RTD for Heterogeneous, Homogeneous, and BPRSNN on SHD (temporal) and CIFAR10-DVS (spatial) task
  • Figure 4: Figure comparing the accuracy. The left y-axis (in red) shows the accuracy of the BPRSNN model with increasing training epochs. The right y-axis shows the RTD between HL3 and BPL3 after each training epoch for the BPRSNN model with the trained HRSNN model. The green horizontal line shows the accuracy of the HRSNN model (80.49%), and the vertical line shows the point where the accuracies of the two models are equal.