Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hybrid Spiking Neural Network -- Transformer Video Classification Model

Aaron Bateni

TL;DR

This work tackles time-series classification using a cortical-column-inspired hybrid spiking neural network (SNN) that processes temporal information via spike timing. It combines unsupervised spike-timing-dependent plasticity (STDP) in an initial stage with reward-modulated STDP in a two-class cortical-column final layer, guided by multiple spike-encoding schemes and a two-step input pipeline (video→images→spike trains). The authors present concrete encoding methods, a biologically inspired network with excitatory/inhibitory connections, and dynamic reward strategies to drive learning, validating the approach on a small sentiment-like dataset derived from video-derived text representations and providing public code. The results suggest potential advantages for small-scale, temporal-aware classification with low power and parallel processing benefits, while highlighting training-time challenges that motivate future hardware and algorithmic optimizations.

Abstract

In recent years, Spiking Neural Networks (SNNs) have gathered significant interest due to their temporal understanding capabilities. This work introduces, to the best of our knowledge, the first Cortical Column like hybrid architecture for the Time-Series Data Classification Task that leverages SNNs and is inspired by the brain structure, inspired from the previous hybrid models. We introduce several encoding methods to use with this model. Finally, we develop a procedure for training this network on the training dataset. As an effort to make using these models simpler, we make all the implementations available to the public.

Hybrid Spiking Neural Network -- Transformer Video Classification Model

TL;DR

This work tackles time-series classification using a cortical-column-inspired hybrid spiking neural network (SNN) that processes temporal information via spike timing. It combines unsupervised spike-timing-dependent plasticity (STDP) in an initial stage with reward-modulated STDP in a two-class cortical-column final layer, guided by multiple spike-encoding schemes and a two-step input pipeline (video→images→spike trains). The authors present concrete encoding methods, a biologically inspired network with excitatory/inhibitory connections, and dynamic reward strategies to drive learning, validating the approach on a small sentiment-like dataset derived from video-derived text representations and providing public code. The results suggest potential advantages for small-scale, temporal-aware classification with low power and parallel processing benefits, while highlighting training-time challenges that motivate future hardware and algorithmic optimizations.

Abstract

In recent years, Spiking Neural Networks (SNNs) have gathered significant interest due to their temporal understanding capabilities. This work introduces, to the best of our knowledge, the first Cortical Column like hybrid architecture for the Time-Series Data Classification Task that leverages SNNs and is inspired by the brain structure, inspired from the previous hybrid models. We introduce several encoding methods to use with this model. Finally, we develop a procedure for training this network on the training dataset. As an effort to make using these models simpler, we make all the implementations available to the public.

Paper Structure

This paper contains 35 sections, 8 equations, 14 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Preliminaries
  5. Spiking Neural Networks
  6. Cortical Columns of the Brain
  7. Spike Time-Dependent Plasticity
  8. Network Components
  9. Neuronal Groups
  10. Synaptic Connections
  11. Nature of Neurons and Response Detection
  12. Training Procedure
  13. Spike-Timing-Dependent Plasticity
  14. Reward-Modulated Spike-Timing-Dependent Plasticity
  15. Reward Calculation Formula
  16. ...and 20 more sections

Figures (14)

  • Figure 1: View of the spiking network used for the mentioned classification tasks along with the learning mechanisms applied to each connection
  • Figure 2: An example of sports videos from the Kinetics-400 dataset.
  • Figure 3: A frame from a tennis match video, containing both visual and temporal data.
  • Figure 4: A sample representation of two images from WaterlooDataset, and their respective encodings' heat maps using Time-To-First-Spike encoding. Pixles with values close to 255 fire closer to 0ms.
  • Figure 5: The corresponding spike train encoding to the previous heat maps.
  • ...and 9 more figures