SynA-ResNet: Spike-driven ResNet Achieved through OR Residual Connection

TL;DR

This work tackles the energy inefficiency and reduced sparsity of deep residual spiking neural networks by introducing OR Residual Connection (ORRC) to preserve redundant information and a Synergistic Attention (SynA) mechanism to sharpen backbone features while suppressing shortcut noise. The approach yields fully spike-driven computation and reveals a natural pruning phenomenon where shortcuts drop out during training without sacrificing accuracy, reducing computational load. Across neuromorphic and static datasets, OR-Spiking ResNet with SynA matches or surpasses state-of-the-art residual SNNs while delivering substantial energy savings, up to 28-fold, and very low spike counts, enabling practical edge deployment. The work contributes a novel combination of bitwise residuals and attention in SNNs, provides detailed ablations and energy analyses, and offers actionable guidance for lightweight SNN design and hardware-compatible implementations.

Abstract

Spiking Neural Networks (SNNs) have garnered substantial attention in brain-like computing for their biological fidelity and the capacity to execute energy-efficient spike-driven operations. As the demand for heightened performance in SNNs surges, the trend towards training deeper networks becomes imperative, while residual learning stands as a pivotal method for training deep neural networks. In our investigation, we identified that the SEW-ResNet, a prominent representative of deep residual spiking neural networks, incorporates non-event-driven operations. To rectify this, we propose a novel training paradigm that first accumulates a large amount of redundant information through OR Residual Connection (ORRC), and then filters out the redundant information using the Synergistic Attention (SynA) module, which promotes feature extraction in the backbone while suppressing the influence of noise and useless features in the shortcuts. When integrating SynA into the network, we observed the phenomenon of "natural pruning", where after training, some or all of the shortcuts in the network naturally drop out without affecting the model's classification accuracy. This significantly reduces computational overhead and makes it more suitable for deployment on edge devices. Experimental results on various public datasets confirmed that the SynA-ResNet achieved single-sample classification with as little as 0.8 spikes per neuron. Moreover, when compared to other residual SNN models, it exhibited higher accuracy and up to a 28-fold reduction in energy consumption.

Figures (5)

• Figure 1: The structure and functional mechanism of the OR-Spiking ResNet residual block and SynA. The shading of the neurons represents the magnitude of their firing rate, with darker shades indicating higher firing rates. Similarly, the small squares representing attention weights use darker colors to denote greater attention weights.
• Figure 2: Mainstream Spiking ResNet network architecture. (a)Vanilla Spiking ResNet; (b)MS-ResNet; (c)SEW-ResNet, where Conv and BN represent convolution and Batch Normalization Layer, respectively. $g$ represents the element-wise functions of the aggregated backbone and shortcut.
• Figure 3: Characteristic strength heatmap for two types of residual connections. We take samples with T=1, 8, 16, 24, and 32 and visualize them when the DVS Gesture encoding is T=32, The purple in the figure is the background, the blue part is the target in the input sample, and the color depth in the red part indicates the Spiking intensity emitted by the LIF neurons at different positions output by the first shortcut in the network.
• Figure 4: Characteristic strength heatmap of the SynA effect in both channel and temporal dimensions.
• Figure 5: Thermogram of the changes in firing rate of all neurons in OR-Spiking ResNet with SynA-C added as training rounds increase. For a clearer observation, We created three bar charts to illustrate the changes in the firing rate of LIF neurons in shortcuts as the number of training rounds increased.