Improving the Sparse Structure Learning of Spiking Neural Networks from the View of Compression Efficiency

TL;DR

This work tackles over-parameterization and fixed pruning in deep Spiking Neural Networks by introducing a two-stage dynamic sparse learning framework that aligns sparsity with data compressibility. Stage I uses the PQ index to estimate a per-iteration rewiring ratio, and Stage II executes pruning and regrowth guided by that ratio, enabling sparse training from scratch. The approach demonstrates competitive accuracy at high sparsity and substantially improved compression efficiency across CIFAR10, CIFAR100, and DVS-CIFAR10, with two rewiring strategies (neuron-wise and layer-wise) and Brain-inspired dynamic structure updates. This compression-aware sparsity enables efficient edge and neuromorphic hardware deployment while preserving the benefits of sparse training from scratch.

Abstract

The human brain utilizes spikes for information transmission and dynamically reorganizes its network structure to boost energy efficiency and cognitive capabilities throughout its lifespan. Drawing inspiration from this spike-based computation, Spiking Neural Networks (SNNs) have been developed to construct event-driven models that emulate this efficiency. Despite these advances, deep SNNs continue to suffer from over-parameterization during training and inference, a stark contrast to the brain's ability to self-organize. Furthermore, existing sparse SNNs are challenged by maintaining optimal pruning levels due to a static pruning ratio, resulting in either under- or over-pruning. In this paper, we propose a novel two-stage dynamic structure learning approach for deep SNNs, aimed at maintaining effective sparse training from scratch while optimizing compression efficiency. The first stage evaluates the compressibility of existing sparse subnetworks within SNNs using the PQ index, which facilitates an adaptive determination of the rewiring ratio for synaptic connections based on data compression insights. In the second stage, this rewiring ratio critically informs the dynamic synaptic connection rewiring process, including both pruning and regrowth. This approach significantly improves the exploration of sparse structure training in deep SNNs, adapting sparsity dynamically from the point view of compression efficiency. Our experiments demonstrate that this sparse training approach not only aligns with the performance of current deep SNNs models but also significantly improves the efficiency of compressing sparse SNNs. Crucially, it preserves the advantages of initiating training with sparse models and offers a promising solution for implementing edge AI on neuromorphic hardware.

Figures (4)

• Figure 1: The flowchart of the proposed two-stage sparse structure learning method for SNNs. Stage I involves the typical training process and attempts to identify an appropriate rewiring ratio according to PQ index. Stage II conducts the dynamic sparse structure learning method based on the rewiring ratio in stage I. The iterative learning of the above two stages is employed during the whole training process, thereby implementing the sparse training from scratch for SNNs and enhancing generalization ability of the sparse model.
• Figure 2: An example for training process of our method (a). The performance of the proposed two-stage sparse training method for SNNs, on CIFAR10 (b) and CIFAR100 (c) datasets, in the neuron-wise pruning scope. The bar chart represents the accuracy achieved by the proposed two-stage sparse training method. The solid line reflects the density of synaptic connections in the SNNs model.
• Figure 3: The performance of the proposed two-stage sparse training method for SNNs, on CIFAR10 (a) and CIFAR100 (b) datasets, in the layer-wise pruning scope. The bar chart represents the accuracy achieved by the proposed method. The solid line reflects the density of synaptic connections in the SNNs model. The dashed line is the trend analysis of accuracy using a two-period moving average. This diagram depicts the correlation between the density of the model and the enhancements in performance achieved using our two-stage sparse structure learning technique.
• Figure 4: The ablation experiments of our proposed two-stage sparse training method for SNNs. (a) The accuracy comparison between the gradually sparse training and sparse training from scratch. (b) The connection density comparison between the gradually sparse training and sparse training from scratch.