Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SVFormer: A Direct Training Spiking Transformer for Efficient Video Action Recognition

Liutao Yu, Liwei Huang, Chenlin Zhou, Han Zhang, Zhengyu Ma, Huihui Zhou, Yonghong Tian

TL;DR

This study innovatively proposes the directly trained SVFormer (Spiking Video transFormer) for VAR, which integrates local feature extraction, global self-attention, and the intrinsic dynamics, sparsity, and spike-driven nature of SNNs, to efficiently and effectively extract spatio-temporal features.

Abstract

Video action recognition (VAR) plays crucial roles in various domains such as surveillance, healthcare, and industrial automation, making it highly significant for the society. Consequently, it has long been a research spot in the computer vision field. As artificial neural networks (ANNs) are flourishing, convolution neural networks (CNNs), including 2D-CNNs and 3D-CNNs, as well as variants of the vision transformer (ViT), have shown impressive performance on VAR. However, they usually demand huge computational cost due to the large data volume and heavy information redundancy introduced by the temporal dimension. To address this challenge, some researchers have turned to brain-inspired spiking neural networks (SNNs), such as recurrent SNNs and ANN-converted SNNs, leveraging their inherent temporal dynamics and energy efficiency. Yet, current SNNs for VAR also encounter limitations, such as nontrivial input preprocessing, intricate network construction/training, and the need for repetitive processing of the same video clip, hindering their practical deployment. In this study, we innovatively propose the directly trained SVFormer (Spiking Video transFormer) for VAR. SVFormer integrates local feature extraction, global self-attention, and the intrinsic dynamics, sparsity, and spike-driven nature of SNNs, to efficiently and effectively extract spatio-temporal features. We evaluate SVFormer on two RGB datasets (UCF101, NTU-RGBD60) and one neuromorphic dataset (DVS128-Gesture), demonstrating comparable performance to the mainstream models in a more efficient way. Notably, SVFormer achieves a top-1 accuracy of 84.03% with ultra-low power consumption (21 mJ/video) on UCF101, which is state-of-the-art among directly trained deep SNNs, showcasing significant advantages over prior models.

SVFormer: A Direct Training Spiking Transformer for Efficient Video Action Recognition

TL;DR

This study innovatively proposes the directly trained SVFormer (Spiking Video transFormer) for VAR, which integrates local feature extraction, global self-attention, and the intrinsic dynamics, sparsity, and spike-driven nature of SNNs, to efficiently and effectively extract spatio-temporal features.

Abstract

Video action recognition (VAR) plays crucial roles in various domains such as surveillance, healthcare, and industrial automation, making it highly significant for the society. Consequently, it has long been a research spot in the computer vision field. As artificial neural networks (ANNs) are flourishing, convolution neural networks (CNNs), including 2D-CNNs and 3D-CNNs, as well as variants of the vision transformer (ViT), have shown impressive performance on VAR. However, they usually demand huge computational cost due to the large data volume and heavy information redundancy introduced by the temporal dimension. To address this challenge, some researchers have turned to brain-inspired spiking neural networks (SNNs), such as recurrent SNNs and ANN-converted SNNs, leveraging their inherent temporal dynamics and energy efficiency. Yet, current SNNs for VAR also encounter limitations, such as nontrivial input preprocessing, intricate network construction/training, and the need for repetitive processing of the same video clip, hindering their practical deployment. In this study, we innovatively propose the directly trained SVFormer (Spiking Video transFormer) for VAR. SVFormer integrates local feature extraction, global self-attention, and the intrinsic dynamics, sparsity, and spike-driven nature of SNNs, to efficiently and effectively extract spatio-temporal features. We evaluate SVFormer on two RGB datasets (UCF101, NTU-RGBD60) and one neuromorphic dataset (DVS128-Gesture), demonstrating comparable performance to the mainstream models in a more efficient way. Notably, SVFormer achieves a top-1 accuracy of 84.03% with ultra-low power consumption (21 mJ/video) on UCF101, which is state-of-the-art among directly trained deep SNNs, showcasing significant advantages over prior models.
Paper Structure (23 sections, 13 equations, 4 figures, 3 tables)

This paper contains 23 sections, 13 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. ANNs for VAR
  4. SNNs for VAR
  5. Preliminary and Methodology
  6. Spiking neural networks
  7. The proposed SVFormer
  8. Overview
  9. Patch embedding module
  10. Local feature extractor
  11. Global self-attention module
  12. Local pathway
  13. Classification head
  14. Time-dependent batch normalization
  15. Overall computation process
  16. ...and 8 more sections

Figures (4)

  • Figure 1: (a) The scheme of a spiking neuron, of which the input and output are both binary spikes. (b) Sigmoid function approximates the Heaviside activation function of a spiking neuron, and the derivative of it can be utilized to calculate gradients during backpropagation.
  • Figure 2: (a) The overall structure of the hierarchical SVFormer, which includes four stages and one local pathway in default. (b) Structure of one local feature extractor (LFE). (c) Strcuture of one global self-attention (GSA) module.
  • Figure 3: Average firing rates (a, b) during model inference on the UCF101 dataset, and learned membrane time constants (c, d), for all PLIF layers, showing the trend of both variables as the network goes deeper (a, c), and the corresponding histograms (b, d).
  • Figure 4: (a) An exemplar frame with different level of Gaussian noise ($N(0, \sigma)$). (b) An exemplar frame with different level of salt-and-pepper noise, where $P$ means the sprinkling probability.