Sparse Spiking Neural Network: Exploiting Heterogeneity in Timescales for Pruning Recurrent SNN

TL;DR

A novel Lyapunov Noise Pruning (LNP) algorithm that uses graph sparsification methods and utilizes Lyapunov exponents to design a stable sparse RSNN from a randomly initialized RSNN, and it is shown that the LNP can leverage diversity in neuronal timescales to design a sparse Heterogeneous RSNN (HRSNN).

Abstract

Recurrent Spiking Neural Networks (RSNNs) have emerged as a computationally efficient and brain-inspired learning model. The design of sparse RSNNs with fewer neurons and synapses helps reduce the computational complexity of RSNNs. Traditionally, sparse SNNs are obtained by first training a dense and complex SNN for a target task, and, then, pruning neurons with low activity (activity-based pruning) while maintaining task performance. In contrast, this paper presents a task-agnostic methodology for designing sparse RSNNs by pruning a large randomly initialized model. We introduce a novel Lyapunov Noise Pruning (LNP) algorithm that uses graph sparsification methods and utilizes Lyapunov exponents to design a stable sparse RSNN from a randomly initialized RSNN. We show that the LNP can leverage diversity in neuronal timescales to design a sparse Heterogeneous RSNN (HRSNN). Further, we show that the same sparse HRSNN model can be trained for different tasks, such as image classification and temporal prediction. We experimentally show that, in spite of being task-agnostic, LNP increases computational efficiency (fewer neurons and synapses) and prediction performance of RSNNs compared to traditional activity-based pruning of trained dense models.

Figures (13)

• Figure 1: (a) Concept of HRSNN with variable Neuronal and Synaptic Dynamics (b) Figure showing the task-agnostic pruning and training of the CHRSNN/HRSNN networks using LNP in comparison to the current approach
• Figure 2: Complete flowchart showing the steps for the LNP pruning algorithm and the training methodology to use the pruned HRSNN network
• Figure 3: Comparative Evaluation of Pruning Methods Across Iterations. Figs. (a) and (b) show the evolution of the number of synapses and neurons with the iterations of the LNP and AP algorithms. Fig (c) represents how the RMSE loss changes when the pruned model after each iteration is trained and tested on the Lorenz63 dataset
• Figure 4: Scatter Plot showing Accuracy vs. Avg. SOPs for different pruning methods on CIFAR10. Results for CIFAR100 & Lorenz63 are given in Suppl. Sec. \ref{['sec:results']}
• Figure 5: Plot showing Ablation studies of LNP