Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Efficient Knowledge Transfer Strategy for Spiking Neural Networks from Static to Event Domain

Xiang He, Dongcheng Zhao, Yang Li, Guobin Shen, Qingqun Kong, Yi Zeng

TL;DR

This work tackles the domain mismatch that arises when transferring knowledge from static images to event-based spiking neural networks. It introduces a knowledge transfer loss composed of a domain alignment term and a spatio-temporal regularization term, enabling domain-invariant spatial features while adaptively weighting temporal information via learnable coefficients. A sliding training strategy gradually replaces static inputs with event data, providing stable optimization and smoother knowledge transfer. Across N-Caltech101, CEP-DVS, and N-Omniglot, the approach achieves state-of-the-art results and demonstrates robust gains even with varying amounts of event data, highlighting practical value for neuromorphic applications.

Abstract

Spiking neural networks (SNNs) are rich in spatio-temporal dynamics and are suitable for processing event-based neuromorphic data. However, event-based datasets are usually less annotated than static datasets. This small data scale makes SNNs prone to overfitting and limits their performance. In order to improve the generalization ability of SNNs on event-based datasets, we use static images to assist SNN training on event data. In this paper, we first discuss the domain mismatch problem encountered when directly transferring networks trained on static datasets to event data. We argue that the inconsistency of feature distributions becomes a major factor hindering the effective transfer of knowledge from static images to event data. To address this problem, we propose solutions in terms of two aspects: feature distribution and training strategy. Firstly, we propose a knowledge transfer loss, which consists of domain alignment loss and spatio-temporal regularization. The domain alignment loss learns domain-invariant spatial features by reducing the marginal distribution distance between the static image and the event data. Spatio-temporal regularization provides dynamically learnable coefficients for domain alignment loss by using the output features of the event data at each time step as a regularization term. In addition, we propose a sliding training strategy, which gradually replaces static image inputs probabilistically with event data, resulting in a smoother and more stable training for the network. We validate our method on neuromorphic datasets, including N-Caltech101, CEP-DVS, and N-Omniglot. The experimental results show that our proposed method achieves better performance on all datasets compared to the current state-of-the-art methods. Code is available at https://github.com/Brain-Cog-Lab/Transfer-for-DVS.

An Efficient Knowledge Transfer Strategy for Spiking Neural Networks from Static to Event Domain

TL;DR

This work tackles the domain mismatch that arises when transferring knowledge from static images to event-based spiking neural networks. It introduces a knowledge transfer loss composed of a domain alignment term and a spatio-temporal regularization term, enabling domain-invariant spatial features while adaptively weighting temporal information via learnable coefficients. A sliding training strategy gradually replaces static inputs with event data, providing stable optimization and smoother knowledge transfer. Across N-Caltech101, CEP-DVS, and N-Omniglot, the approach achieves state-of-the-art results and demonstrates robust gains even with varying amounts of event data, highlighting practical value for neuromorphic applications.

Abstract

Spiking neural networks (SNNs) are rich in spatio-temporal dynamics and are suitable for processing event-based neuromorphic data. However, event-based datasets are usually less annotated than static datasets. This small data scale makes SNNs prone to overfitting and limits their performance. In order to improve the generalization ability of SNNs on event-based datasets, we use static images to assist SNN training on event data. In this paper, we first discuss the domain mismatch problem encountered when directly transferring networks trained on static datasets to event data. We argue that the inconsistency of feature distributions becomes a major factor hindering the effective transfer of knowledge from static images to event data. To address this problem, we propose solutions in terms of two aspects: feature distribution and training strategy. Firstly, we propose a knowledge transfer loss, which consists of domain alignment loss and spatio-temporal regularization. The domain alignment loss learns domain-invariant spatial features by reducing the marginal distribution distance between the static image and the event data. Spatio-temporal regularization provides dynamically learnable coefficients for domain alignment loss by using the output features of the event data at each time step as a regularization term. In addition, we propose a sliding training strategy, which gradually replaces static image inputs probabilistically with event data, resulting in a smoother and more stable training for the network. We validate our method on neuromorphic datasets, including N-Caltech101, CEP-DVS, and N-Omniglot. The experimental results show that our proposed method achieves better performance on all datasets compared to the current state-of-the-art methods. Code is available at https://github.com/Brain-Cog-Lab/Transfer-for-DVS.
Paper Structure (45 sections, 10 equations, 9 figures, 8 tables, 1 algorithm)

This paper contains 45 sections, 10 equations, 9 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Domain Adaptation Using Static Data.
  4. Event-Based Data Augmentation.
  5. SNN Efficient Training.
  6. Preliminaries
  7. Neuron Model.
  8. Processing of Neuromorphic Data.
  9. Methods
  10. Domain Mismatch
  11. Knowledge Transfer Loss Function
  12. Domain Alignment Loss.
  13. Spatio-Temporal Regularization.
  14. Sliding Training Strategy
  15. Experiments
  16. ...and 30 more sections

Figures (9)

  • Figure 1: Top: Visualization of network representation similarity. The left-left side panel shows the cross-layer heatmap, while the right side panel shows the diagonal of the cross-layer heatmap. Middle: Visualization of the distribution of membrane potentials. The left and right figures show the results of the membrane potential distribution based on static data and event data training, respectively. Bottom: Accuracy curves when pre-trained model on static data, with fine-tuning on event data. The latter half of the epochs is shown.
  • Figure 2: Proposed knowledge transfer framework for spiking neural network. Static image and event data are input simultaneously and share the network weights except for the last layer. The membrane potential of the neurons in the second-last layer is used to calculate the knowledge transfer loss. MP node in last layer means using membrane potential output.
  • Figure 3: Performance of baseline and knowledge transfer loss methods on the N-Caltech101 dataset.
  • Figure 4: The loss landscape of visualization of our method and baseline on N-Caltech101 and CEP-DVS dataset.
  • Figure 5: Class Activation Mapping of Caltech101 and N-Caltech101. Three categories are selected for display, the top row under each category represents static images, and the bottom row represents event data integrated into frames. The three columns from left to right represent the results of original picture, baseline and our method, respectively.
  • ...and 4 more figures