Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Efficient Deep Spiking Neural Networks Construction with Spiking Activity based Pruning

Yaxin Li, Qi Xu, Jiangrong Shen, Hongming Xu, Long Chen, Gang Pan

TL;DR

This paper addresses redundancy in deep spiking neural networks by introducing Spiking Channel Activity-based (SCA) pruning, a structured, dynamic sparsity framework that prunes and regenerates convolutional channels during training based on spiking activity. By combining a channel-importance score with a prune-and-regrow rule governed by sparsity targets and BN gamma gradients, SCA yields lightweight networks with minimal accuracy loss across CIFAR-10/100 and DVS-CIFAR10, while reducing parameters and SynOps. The approach is applicable to various architectures (e.g., VGG, ResNet, PreResNet) and aligns with hardware deployment needs, offering improved energy efficiency and hardware friendliness. Overall, SCA demonstrates that structured dynamic sparse learning can effectively adapt deep SNNs to target tasks and resource-constrained environments, advancing practical neuromorphic computing.

Abstract

The emergence of deep and large-scale spiking neural networks (SNNs) exhibiting high performance across diverse complex datasets has led to a need for compressing network models due to the presence of a significant number of redundant structural units, aiming to more effectively leverage their low-power consumption and biological interpretability advantages. Currently, most model compression techniques for SNNs are based on unstructured pruning of individual connections, which requires specific hardware support. Hence, we propose a structured pruning approach based on the activity levels of convolutional kernels named Spiking Channel Activity-based (SCA) network pruning framework. Inspired by synaptic plasticity mechanisms, our method dynamically adjusts the network's structure by pruning and regenerating convolutional kernels during training, enhancing the model's adaptation to the current target task. While maintaining model performance, this approach refines the network architecture, ultimately reducing computational load and accelerating the inference process. This indicates that structured dynamic sparse learning methods can better facilitate the application of deep SNNs in low-power and high-efficiency scenarios.

Towards Efficient Deep Spiking Neural Networks Construction with Spiking Activity based Pruning

TL;DR

This paper addresses redundancy in deep spiking neural networks by introducing Spiking Channel Activity-based (SCA) pruning, a structured, dynamic sparsity framework that prunes and regenerates convolutional channels during training based on spiking activity. By combining a channel-importance score with a prune-and-regrow rule governed by sparsity targets and BN gamma gradients, SCA yields lightweight networks with minimal accuracy loss across CIFAR-10/100 and DVS-CIFAR10, while reducing parameters and SynOps. The approach is applicable to various architectures (e.g., VGG, ResNet, PreResNet) and aligns with hardware deployment needs, offering improved energy efficiency and hardware friendliness. Overall, SCA demonstrates that structured dynamic sparse learning can effectively adapt deep SNNs to target tasks and resource-constrained environments, advancing practical neuromorphic computing.

Abstract

The emergence of deep and large-scale spiking neural networks (SNNs) exhibiting high performance across diverse complex datasets has led to a need for compressing network models due to the presence of a significant number of redundant structural units, aiming to more effectively leverage their low-power consumption and biological interpretability advantages. Currently, most model compression techniques for SNNs are based on unstructured pruning of individual connections, which requires specific hardware support. Hence, we propose a structured pruning approach based on the activity levels of convolutional kernels named Spiking Channel Activity-based (SCA) network pruning framework. Inspired by synaptic plasticity mechanisms, our method dynamically adjusts the network's structure by pruning and regenerating convolutional kernels during training, enhancing the model's adaptation to the current target task. While maintaining model performance, this approach refines the network architecture, ultimately reducing computational load and accelerating the inference process. This indicates that structured dynamic sparse learning methods can better facilitate the application of deep SNNs in low-power and high-efficiency scenarios.
Paper Structure (15 sections, 4 equations, 4 figures, 1 table, 1 algorithm)

This paper contains 15 sections, 4 equations, 4 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminary
  4. Methodology
  5. The SCA Structure Learning Framework
  6. Channel Importance Score
  7. Structure Learning Rule
  8. Various Network Architectures
  9. Experiments
  10. Experiment Settings.
  11. Evaluation on different datasets.
  12. Analysis of Network Structure Learning.
  13. Analysis of Ablation Experiments.
  14. Comparison with Other Methods.
  15. Conclusion

Figures (4)

  • Figure 1: The schematic illustration of the SCA structure learning framework. (a) The proposed framework follows structure learning rules to adaptively learn lightweight network models during training. (b) The proposed framework is applicable to various network structures like pre-activation, post-activation, etc.
  • Figure 2: The performance of the SCA structure learning framework.
  • Figure 3: Analysis of structural changes in the SCA framework.
  • Figure 4: Analysis of the effectiveness of the SCA structure learning framework.