Biologically-Informed Excitatory and Inhibitory Balance for Robust Spiking Neural Network Training

TL;DR

This study addresses robust training of biologically constrained spiking neural networks by systematically varying excitatory/inhibitory balance ($E:I$) and initial activity, evaluating performance on Fashion-MNIST and SHD with surrogate-gradient learning. The authors show that low initial firing rates and biologically realistic $E:I$ ratios, particularly $80:20$, yield strong accuracy and robustness to noise, with inhibitory diversity (as reflected in spike-train distances) correlating with improved performance. Using neuroscience-inspired metrics such as the Van Rossum distance, they demonstrate that successful training involves differentiated inhibitory activity and stable, differentiable excitatory dynamics, even under noisy weight updates. The findings have implications for energy-efficient SNN design and neuromorphic hardware, suggesting initialization regimes and $E:I$ configurations that balance learning performance with hardware stochasticity, and pointing toward future work with more complex neuron models and multimodal data.

Abstract

Spiking neural networks drawing inspiration from biological constraints of the brain promise an energy-efficient paradigm for artificial intelligence. However, challenges exist in identifying guiding principles to train these networks in a robust fashion. In addition, training becomes an even more difficult problem when incorporating biological constraints of excitatory and inhibitory connections. In this work, we identify several key factors, such as low initial firing rates and diverse inhibitory spiking patterns, that determine the overall ability to train spiking networks with various ratios of excitatory to inhibitory neurons on AI-relevant datasets. The results indicate networks with the biologically realistic 80:20 excitatory:inhibitory balance can reliably train at low activity levels and in noisy environments. Additionally, the Van Rossum distance, a measure of spike train synchrony, provides insight into the importance of inhibitory neurons to increase network robustness to noise. This work supports further biologically-informed large-scale networks and energy efficient hardware implementations.

Figures (5)

• Figure 1: Network protocol overview and training results. a) The spiking networks consist of three unique fully connected layers. The input layer consists of excitatory spike generators. The hidden layer consists of LIF neurons that have either excitatory or inhibitory weight connections to the output layer of leaky-integrators. b) Sample Fashion-MNIST activity of the network starting with the input layer. Spikes of the hidden layer are indicated with a raster plot inset. Hidden layer activity is then propagated to the output where the highest voltage trace corresponds to the networks chosen class (drawn in black). c) Network ability to train on the Fashion-MNIST dataset relative to their initial hidden layer firing rate. The different activity levels correspond to the different initial weight distributions. Every trial corresponds to an individual point on its respective graph. Training consisted of 4 different E:I ratios (left to right 50:50, 80:20, 95:5, 100:0). The black line at 86% accuracy shows the accuracy of previous training of a network of the same size but with unbounded weights. d) Network ability to train on the SHD dataset across a range of initial hidden layer firing rates. From left to right 4 E:I ratios: 50:50, 80:20, 95:5, 100:0. The black line at 48% accuracy represents a network of the same size but with unbounded weights.
• Figure 2: Accuracy of networks with noisy weight updates. a) Weight updates where the pre-update distribution determined the gradient for the update. The gradient and noise distributions were then combined with the weight distribution (this example was an $\sigma_{noise}=0.2\sigma_{init}$) and bounded to get the post-update weight distribution. b) Average accuracy of the final 10 of 30 epochs of training with 16 repeat trials for each E:I ratio and noise level combinations. The average across repeat trials is represented with corresponding lines. Each trial is represented with individual points. Average accuracy across trials compared to the 100:0 E:I with positive accuracy differences indicating an improvement over the 100:0 networks at three different $\sigma_{noise}$ levels: (c) $0.2\sigma_{init}$, (d) $0.4\sigma_{init}$, and (e) $0.6\sigma_{init}$. T-tests were performed to calculate the difference between each E:I ratio and the baseline 100:0 network with * indicating p $<$ 0.05, ** for p $<$ 0.01, and *** for p $<$ 0.001.
• Figure 3: Fashion-MNIST class accuracy relative to the amount of activity and percent of activity that is excitatory across training. Separation is observed between (a) high accuracy networks ($>$50% maximum overall accuracy) at lower activity levels, and (b) low accuracy networks at high activity levels (first column). (Second to fifth column) The initial network, after a single epoch of training, after ten epochs of training, and after training is complete (thirty epochs). Each network is represented by 10 points (for the 10 classes of the dataset) and all networks tested from Figure 1 with E:I ratios of 50:50, 80:20, and 95:5.
• Figure 4: Representative activity of three 50:50 Fashion-MNIST trials. The examples cover (a) low initial activity, (b) moderate initial activity, and (c) higher initial activity. The average number of excitatory (first row) and inhibitory (second row) spikes per image denoting the number of spikes generated by the hidden layer on average given any input image in the Fashion-MNIST dataset. Each line represents a different class with the black line noting the average across all classes. The percentage of excitatory spikes across training (third row) indicates the initial increase in proportional excitatory activity, followed by a decrease for the remainder of training in the low and moderate initial activity conditions. The adjustment of activity from the network training perspective, derived from the weight distributions before (fourth row) and after (fifth row) training. The accuracy convergence curve of each individual trial is included as the inset of the post-training weights.
• Figure 5: Average Van Rossum distances between hidden layer neuron pairs trained on the Fashion-MNIST (a-c) and SHD (d-f) datasets. Rows correspond to different E:I ratios 50:50 (a,d), 80:20 (b,e), 95:5 (c,f). The average distances between neuron pairs: excitatory-excitatory (E-E), excitatory-inhibitory (E-I), and the inhibitory-inhibitory (I-I) pairs. The distance is averaged across all cases. Each network tested in Figure 1 is represented by a single average value for each of the three categories and make the distributions in each subfigure. The initial networks spread a range of distances based on their varied weight initializations but show equal distributions across the three categories. The successfully trained networks from the Fashion-MNIST trials are defined as networks with peak overall accuracy $>$50%, and have a lower distribution of distances, while the networks that fail to reach 50% accuracy are typically in a higher range of activity, with the difference between failure and success being stronger in higher E:I ratios. Successfully trained networks from the SHD trials are defined as networks with average accuracy $>$30% for the last 25 epochs. Using the Kruskal-Wallis test, the three categories are calculated to be statistically significantly different with p-values $<$0.0001 for the successful 80:20 and 95:5 networks, with I-I distances being the largest, and E-E distances being the lowest.