A flexible framework for structural plasticity in GPU-accelerated sparse spiking neural networks

TL;DR

The paper introduces a flexible, GPU-accelerated framework for structural plasticity in sparse spiking neural networks, built on GeNN, to enable runtime addition and removal of synapses with minimal data movement. It combines a modular custom connectivity update system, mlGeNN for high-level training with e-prop and DEEP R, and two validated models: a highly sparse recurrent classifier and a topographic map network. Empirical results show that sparse DEEP R-trained classifiers can match dense models with dramatically fewer connections and substantial training speedups (up to 15×, with overhead <1%), while the topographic map demonstrates scalable, faster-than-realtime simulations and insightful connectivity evolution. Overall, the framework advances efficient exploration of structural plasticity mechanisms and their application to neuromorphic and spike-based ML tasks, with potential extensions to broader hardware platforms and learning rules.

Abstract

The majority of research in both training Artificial Neural Networks (ANNs) and modeling learning in biological brains focuses on synaptic plasticity, where learning equates to changing the strength of existing connections. However, in biological brains, structural plasticity - where new connections are created and others removed - is also vital, not only for effective learning but also for recovery from damage and optimal resource usage. Inspired by structural plasticity, pruning is often used in machine learning to remove weak connections from trained models to reduce the computational requirements of inference. However, the machine learning frameworks typically used for backpropagation-based training of both ANNs and Spiking Neural Networks (SNNs) are optimized for dense connectivity, meaning that pruning does not help reduce the training costs of ever-larger models. The GeNN simulator already supports efficient GPU-accelerated simulation of sparse SNNs for computational neuroscience and machine learning. Here, we present a new flexible framework for implementing GPU-accelerated structural plasticity rules and demonstrate this first using the e-prop supervised learning rule and DEEP R to train efficient, sparse SNN classifiers and then, in an unsupervised learning context, to learn topographic maps. Compared to baseline dense models, our sparse classifiers reduce training time by up to 10x while the DEEP R rewiring enables them to perform as well as the original models. We demonstrate topographic map formation in faster-than-realtime simulations, provide insights into the connectivity evolution, and measure simulation speed versus network size. The proposed framework will enable further research into achieving and maintaining sparsity in network structure and neural communication, as well as exploring the computational benefits of sparsity in a range of neuromorphic applications.

Figures (9)

• Figure 1: Illustration of data structures used to implement sparse connectivity in GeNN. Five presynaptic neurons are connected to four postsynaptic neurons using sparse connectivity with a maximum row length of three. This is represented in memory as a ragged matrix, where each row n contains the indices of the target neurons of pre-synaptic neuron n. Spikes are propagated through ragged matrix connectivity by parallelizing across the columns as indicated by the snaking lines.
• Figure 2: Illustration of data structures used for the DEEP R implementation. Five presynaptic neurons are connected to four postsynaptic neurons using sparse connectivity with a maximum row length of three. Each panel shows a network sketch, the ragged matrix and two bitfields -- one mirroring the connectivity to provide a faster lookup and the other holding the fixed sign of each potential connection. (A) Initial state. Weights whose sign no longer matches the sign bit are highlighted in green. (B) Elimination pass. Connections highlighted in (A) are removed, the number of connections in each row made 'dormant' are counted and the bitfield is updated. (C) Formation pass. Yellow highlighting indicates new the randomly added connections.
• Figure 3: Illustration of data structures used for one custom connectivity update according to the synaptic rewiring model for topographic maps. Each panel shows a network sketch, the ragged matrix (number of presynaptic neurons $\times$ row length), the bitfield (number of presynaptic neurons $\times$ number of postsynaptic neurons), and pseudocode. (A) Initial state. Five presynaptic neurons, sources $s_{j}$ with $0\leq j<5,$ are connected to four postsynaptic neurons, targets $t_{i}$ with $0\leq i<4$, using sparse connectivity with a maximum row length of three. Weights below the threshold $g_{\theta}$ are indicated in light red, others in light blue. A synapse slot from $s_{j}$ to $t_{i}$ is here referred to as $\left(s_{j},t_{i}\right)$. The bitfield is cleared. (B) Host assigns six rewiring attempts randomly to three presynaptic neurons ($s_{0}$, $s_{2}$, $s_{4}$). The presynaptic neurons then in parallel select their postsynaptic targets randomly and set the corresponding bits as candidates for rewiring attempts in the bitfield. Four existing synapses are selected for elimination (green; $\left(s_{0},t_{0}\right)$, $\left(s_{0},t_{2}\right)$, $\left(s_{4},t_{2}\right)$, $\left(s_{4},t_{3}\right)$) and two empty synapse slots for formation (yellow; $\left(s_{2},t_{2}\right),\left(s_{4},t_{0}\right)$). (C) Elimination attempts. $\left(s_{0},t_{0}\right)$ has a low elimination probability because of the strong weight (light blue) and remains, the three others have a higher elimination probability because of the lower weights (light red); $\left(s_{0},t_{2}\right)$ gets eliminated reducing the row length of $s_{0}$ in the ragged matrix from two to one; $\left(s_{4},t_{2}\right)$ gets eliminated, and the emerging gap in the row of $s_{4}$ is filled by inserting the last element $t_{3}$; $\left(s_{4},t_{3}\right)$ remains. The elimination bits get cleared in the bitfield. (D) Formation attempts. Assuming a spatial proximity of $s_{2}$ and $t_{2}$ and therefore a high formation probability, a new synapse (purple) is formed at $\left(s_{2},t_{2}\right)$, increasing the row length of $s_{2}$ from zero to one, but not at $\left(s_{4},t_{0}\right)$ because $s_{4}$ and $t_{0}$ are farther apart. The formation bits get cleared in the bitfield.
• Figure 4: Topographic map model. (A) Network sketch indicating initial connections and ideal location of the preferred location for one neuron. (B) Spatially correlated input firing rates; the center (x-marker) changes in regular time intervals. Connection probabilities and weights for (C) feed-forward and (D) lateral connections, averaged over displacements for each grid position.
• Figure 5: DVS gesture classification results.(A) Accuracy. Bar heights show mean accuracy and error bars standard deviations, all calculated over five repeats with different seeds. The dashed line shows the mean test accuracy of the baseline dense network. (B) Training time. The sparse models shown have the same sparsity in $W^{\textrm{in}}$and $W^{\textrm{rec}}$.