Developmental Plasticity-inspired Adaptive Pruning for Deep Spiking and Artificial Neural Networks

TL;DR

This paper proposed a developmental plasticity-inspired adaptive pruning (DPAP) method, with inspiration from the adaptive developmental pruning of dendritic spines, synapses, and neurons according to the “use it or lose it, gradually decay” principle.

Abstract

Developmental plasticity plays a prominent role in shaping the brain's structure during ongoing learning in response to dynamically changing environments. However, the existing network compression methods for deep artificial neural networks (ANNs) and spiking neural networks (SNNs) draw little inspiration from brain's developmental plasticity mechanisms, thus limiting their ability to learn efficiently, rapidly, and accurately. This paper proposed a developmental plasticity-inspired adaptive pruning (DPAP) method, with inspiration from the adaptive developmental pruning of dendritic spines, synapses, and neurons according to the ``use it or lose it, gradually decay" principle. The proposed DPAP model considers multiple biologically realistic mechanisms (such as dendritic spine dynamic plasticity, activity-dependent neural spiking trace, and local synaptic plasticity), with additional adaptive pruning strategy, so that the network structure can be dynamically optimized during learning without any pre-training and retraining. Extensive comparative experiments show consistent and remarkable performance and speed boost with the extremely compressed networks on a diverse set of benchmark tasks for deep ANNs and SNNs, especially the spatio-temporal joint pruning of SNNs in neuromorphic datasets. This work explores how developmental plasticity enables complex deep networks to gradually evolve into brain-like efficient and compact structures, eventually achieving state-of-the-art (SOTA) performance for biologically realistic SNNs.

Figures (3)

• Figure 1: The procedure of DPAP method. The SNN structure (top block) consists of convolutional layers and fully connected layers. Pruning criteria (middle block) contains trace-based BCM plasticity for synapses and dendritic spine plasticity for neurons. Adaptive pruning (bottom block) gradually prunes decayed synapses and neurons according to survival function. The orange graph represents the survival function of the pruned synapse, and the red graph represents the survival function of pruned neurons.
• Figure 2: The effectiveness of introducing DPAP to DSNNs and DNNs.(A) to (C): The test accuracy, convergence speed and energy consumption achieved with and without DPAP, respectively. (D) to (H): Under different pruning rates, the accuracy changes with the iteration process for different datasets. (I) to (M): The test accuracy achieved by DPAP with different pruning rates for different datasets.
• Figure 3: Analyses and visualization of the biological plausibility of DPAP.(A): Visualization of temporal spikes, spiking traces, and the importance of spatial pruning computed from temporal spiking traces. Take for example the first four channels of the first and second convolutional layers of the N-MNIST network with the timesteps of 20. (B,C): The retained neurons and the average importance of all neurons (or synapses) The white squares represent pruned neurons and synapses. (D,E): During the 8 epochs before the neurons or synapses are pruned, the changing of survival function $F_{D}$ and $F_{BCM}$. (F): The changing of network parameters during learning. (G): Test accuracy comparison of different synaptic plasticity used.