Training Deep Normalization-Free Spiking Neural Networks with Lateral Inhibition

TL;DR

This paper presents a normalization-free learning framework for deep spiking neural networks by embedding explicit excitatory–inhibitory (E-I) circuits that implement subtractive and divisive lateral inhibition. To enable stable training, the authors introduce E-I Init, a dynamic initialization that balances E and I currents and sets initial gain, and E-I Prop, a training mechanism that decouples forward and backward passes with adaptive divisive inhibition, STE, and gradient scaling. Across static and neuromorphic benchmarks, DeepEISNN with E-I circuits achieves competitive performance without batch normalization, including $92.05\%$ top-1 on CIFAR-10 and $94.86\%$ on DVS-Gesture, while ablations confirm the necessity of each component. The results offer a biologically grounded alternative to normalization in deep SNNs and provide insights into cortical E-I dynamics, with implications for neuromorphic hardware compatibility and further neuroscience-inspired computation. The work thus advances both practical SNN training and computational neuroscience by delivering a robust, normalization-free framework that preserves biological plausibility while maintaining high performance.

Abstract

Spiking neural networks (SNNs) have garnered significant attention as a central paradigm in neuromorphic computing, owing to their energy efficiency and biological plausibility. However, training deep SNNs has critically depended on explicit normalization schemes, leading to a trade-off between performance and biological realism. To resolve this conflict, we propose a normalization-free learning framework that incorporates lateral inhibition inspired by cortical circuits. Our framework replaces the traditional feedforward SNN layer with a circuit of distinct excitatory (E) and inhibitory (I) neurons that captures the features of the canonical architecture of cortical E-I circuits. The circuit dynamically regulates neuronal activity through subtractive and divisive inhibition, which respectively control the activity and the gain of excitatory neurons. To enable and stabilize end-to-end training of the biologically constrained SNN, we propose two key techniques: E-I Init and E-I Prop. E-I Init is a dynamic parameter initialization scheme that balances excitatory and inhibitory inputs while performing gain control. E-I Prop decouples the backpropagation of the E-I circuits from the forward pass and regulates gradient flow. Experiments across multiple datasets and network architectures demonstrate that our framework enables stable training of deep normalization-free SNNs with biological realism and achieves competitive performance without resorting to explicit normalization schemes. Therefore, our work not only provides a solution to training deep SNNs but also serves as a computational platform for further exploring the functions of E-I interactions in large-scale cortical computation.

Figures (8)

• Figure 1: The proposed feedforward E-I circuit (left), compared with normalization-equipped architecture (right, taking BN as an example). Neurons in layer $l-1$ and $l+1$ are not shown. In the E-I circuit, excitatory and inhibitory neurons are explicitly separated. In contrast, the architecture with explicit normalization between layers does not distinguish neuron types.
• Figure 2: An overview of the proposed framework. E-I Init enables effective learning from the very beginning through a dynamic parameter initialization scheme. E-I Prop then ensures stable end-to-end training by regulating the forward and backward passes. For the sake of brevity, our discussion mainly focuses on a fully connected architecture. Extension of our method to convolutional neural networks (CNNs) is detailed in Appendix \ref{['appendix_cnn']}.
• Figure 3: Mechanism of adaptive stabilization and STE. Forward (bottom-up): The adaptive stabilization handles numerical instability by dynamically replacing zero elements in the denominator with the smallest positive value in the sample, preserving the maximal dynamic range. Backward (top-down): The STE allows gradients to bypass the replacement operation, treating it as an identity function to ensure stable learning.
• Figure 4: Distributions of the integrated input currents in the first, third and fifth layers of our model before and after training.
• Figure 5: Empirical analysis of gradient norms at initialization for convolutional layers in VGG-8, without gradient scaling. The gradients for ${\bm{W}}_{\mathrm{EI}}$ (red bars) are orders of magnitude larger than those for ${\bm{W}}_{\mathrm{EE}}$, ${\bm{W}}_{\mathrm{IE}}$, and gain parameters (blue bars), consistent with the theoretical analysis.