Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Delays Through Gradients and Structure: Emergence of Spatiotemporal Patterns in Spiking Neural Networks

Balázs Mészáros, James Knight, Thomas Nowotny

TL;DR

The potential of combining delay learning with dynamic pruning to develop efficient SNN models for temporal data processing is demonstrated and the preservation of spatio-temporal dynamics throughout pruning and rewiring highlights the robustness of these features, providing a solid foundation for future neuromorphic computing applications.

Abstract

We present a Spiking Neural Network (SNN) model that incorporates learnable synaptic delays through two approaches: per-synapse delay learning via Dilated Convolutions with Learnable Spacings (DCLS) and a dynamic pruning strategy that also serves as a form of delay learning. In the latter approach, the network dynamically selects and prunes connections, optimizing the delays in sparse connectivity settings. We evaluate both approaches on the Raw Heidelberg Digits keyword spotting benchmark using Backpropagation Through Time with surrogate gradients. Our analysis of the spatio-temporal structure of synaptic interactions reveals that, after training, excitation and inhibition group together in space and time. Notably, the dynamic pruning approach, which employs DEEP R for connection removal and RigL for reconnection, not only preserves these spatio-temporal patterns but outperforms per-synapse delay learning in sparse networks. Our results demonstrate the potential of combining delay learning with dynamic pruning to develop efficient SNN models for temporal data processing. Moreover, the preservation of spatio-temporal dynamics throughout pruning and rewiring highlights the robustness of these features, providing a solid foundation for future neuromorphic computing applications.

Learning Delays Through Gradients and Structure: Emergence of Spatiotemporal Patterns in Spiking Neural Networks

TL;DR

The potential of combining delay learning with dynamic pruning to develop efficient SNN models for temporal data processing is demonstrated and the preservation of spatio-temporal dynamics throughout pruning and rewiring highlights the robustness of these features, providing a solid foundation for future neuromorphic computing applications.

Abstract

We present a Spiking Neural Network (SNN) model that incorporates learnable synaptic delays through two approaches: per-synapse delay learning via Dilated Convolutions with Learnable Spacings (DCLS) and a dynamic pruning strategy that also serves as a form of delay learning. In the latter approach, the network dynamically selects and prunes connections, optimizing the delays in sparse connectivity settings. We evaluate both approaches on the Raw Heidelberg Digits keyword spotting benchmark using Backpropagation Through Time with surrogate gradients. Our analysis of the spatio-temporal structure of synaptic interactions reveals that, after training, excitation and inhibition group together in space and time. Notably, the dynamic pruning approach, which employs DEEP R for connection removal and RigL for reconnection, not only preserves these spatio-temporal patterns but outperforms per-synapse delay learning in sparse networks. Our results demonstrate the potential of combining delay learning with dynamic pruning to develop efficient SNN models for temporal data processing. Moreover, the preservation of spatio-temporal dynamics throughout pruning and rewiring highlights the robustness of these features, providing a solid foundation for future neuromorphic computing applications.
Paper Structure (8 sections, 4 equations, 3 figures)

This paper contains 8 sections, 4 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Methods
  3. Results
  4. Spatio-temporal receptive fields of trained networks
  5. Spatio-temporal autocorrelation of learned receptive fields
  6. Dynamic pruning
  7. Ablation study
  8. Discussion

Figures (3)

  • Figure 1: (A) Toy example of how the spatio-temporal receptive fields were generated. $W$ denotes the synaptic strength, $D$ denotes the synaptic delay. (B) The receptive field with the highest observed Moran's I value prior to training (left) and after training (middle) in a dense network. The distributions (right) show all observed Moran's I values. (C) The receptive field with the highest observed Moran's I value prior to training (left) and after training (middle) in a sparse network. The distributions (right) show all observed Moran's I values.
  • Figure 2: Types of learning. Delays are indicated by the thickness of the connections, and red is used to highlight the changes.
  • Figure 3: Comparing the effects of structure learning and delay learning. Learning the structure does not make a huge difference with lower sparsity, but as it increases the benefit becomes clear. This might not be that surprising, since these methods were built to train highly sparse models. Delay learning improves the results with a fixed connectivity matrix, as was also shown in hammouamri2023learning. The benefits of delay learning are not obvious when the structure is learnt.