Brain-Inspired Efficient Pruning: Exploiting Criticality in Spiking Neural Networks

TL;DR

This paper first explains criticality in SNNs from the perspective of maximizing feature information entropy, and proposes a low‐cost metric to assess neuron criticality in feature transmission and designs a pruning‐regeneration method that incorporates this criticality into the pruning process.

Abstract

Spiking Neural Networks (SNNs) have gained significant attention due to the energy-efficient and multiplication-free characteristics. Despite these advantages, deploying large-scale SNNs on edge hardware is challenging due to limited resource availability. Network pruning offers a viable approach to compress the network scale and reduce hardware resource requirements for model deployment. However, existing SNN pruning methods cause high pruning costs and performance loss because they lack efficiency in processing the sparse spike representation of SNNs. In this paper, inspired by the critical brain hypothesis in neuroscience and the high biological plausibility of SNNs, we explore and leverage criticality to facilitate efficient pruning in deep SNNs. We firstly explain criticality in SNNs from the perspective of maximizing feature information entropy. Second, We propose a low-cost metric for assess neuron criticality in feature transmission and design a pruning-regeneration method that incorporates this criticality into the pruning process. Experimental results demonstrate that our method achieves higher performance than the current state-of-the-art (SOTA) method with up to 95.26\% reduction of pruning cost. The criticality-based regeneration process efficiently selects potential structures and facilitates consistent feature representation.

Figures (6)

• Figure 1: Schematic view of Pruning-Regeneration Process. Faded structures and connections with dotted lines are pruned. The orange ones with higher criticality scores are regenerated in our method. (a) Unstructured pruning via global magnitude pruning criterion and regeneration based on criticality. (b) Structured pruning via the scale factor $\gamma$ (BN layer parameter) and regeneration based on criticality.
• Figure 2: Comparison of regeneration methods based on criticality (ours), gradient, random, and GraNet liu2021sparse. The two dashed lines represent the performance of the dense model and the baseline, respectively. (a) Accuracy of VGG16 with 90% sparsity for CIFAR100. (b) Accuracy of ResNet19 with 90% sparsity for CIFAR100.
• Figure 3: (a) Performance of dense model, the baseline, and the proposed method under different surrogate functions on VGG16 for CIFAR100. (b) Mean importance of non-overlapping channels between our method and the baseline for structured pruning after fine-tuning.
• Figure 4: Comparison of the difference in feature extraction of models pruned by our method and the baseline. Our method achieves more uniform feature representations. (a) Comparison of mean of intra-cluster variance between proposed method and the baseline for VGG16 after unstructured pruning (U-Pruning) and ResNet19 after structured pruning (S-Pruning). (b) Average cosine similarity between the class feature map between the training and test dataset for VGG16 after unstructured pruning (U-Pruning) and ResNet19 after structured pruning (S-Pruning). We compare the results between the proposed method and the baseline. (c) Train accuracy and test accuracy on VGG16 for CIFAR100 after unstructured pruning using the proposed method and the baseline. (d) Train accuracy and test accuracy on ResNet19 for CIFAR100 after structured pruning using the proposed method and the baseline.
• Figure 5: Comparison of the difference in feature extraction of models pruned by our method and the baseline. Our method achieves more uniform feature representations. (a) Intra-cluster variance of VGG-16 through unstructured pruning. (b) Intra-cluster variance of ResNet-19 through structured pruning. (c) Cosine similarity of the means of features for each class between the training and test dataset on VGG-16 through unstructured pruning. (d) Cosine similarity of the means of features for each class between the training and test dataset on ResNet-19 through structured pruning.