ASRC-SNN: Adaptive Skip Recurrent Connection Spiking Neural Network

TL;DR

Treating recurrent structure and neuron dynamics jointly reveals a gradient vanishing/exploding problem along the temporal dimension in RSNNs. The authors introduce Skip Recurrent Connections (SRC) to mitigate this issue and Adaptive Skip Recurrent Connections (ASRC) to learn layer-specific skip spans using a temperature-scaled Softmax kernel, enabling dynamic selection among multiple temporal offsets. Across S-MNIST, PS-MNIST, SSC, GSC, SRC-SNN improves long-term temporal modeling while ASRC-SNN achieves state-of-the-art results and greater robustness, including under challenging conditions and with sparse connectivity. The work offers a practical, hardware-friendly approach to robust temporal memory in SNNs and motivates future extensions to learn discrete time (and possibly space) positions for skip connections.

Abstract

In recent years, Recurrent Spiking Neural Networks (RSNNs) have shown promising potential in long-term temporal modeling. Many studies focus on improving neuron models and also integrate recurrent structures, leveraging their synergistic effects to improve the long-term temporal modeling capabilities of Spiking Neural Networks (SNNs). However, these studies often place an excessive emphasis on the role of neurons, overlooking the importance of analyzing neurons and recurrent structures as an integrated framework. In this work, we consider neurons and recurrent structures as an integrated system and conduct a systematic analysis of gradient propagation along the temporal dimension, revealing a challenging gradient vanishing problem. To address this issue, we propose the Skip Recurrent Connection (SRC) as a replacement for the vanilla recurrent structure, effectively mitigating the gradient vanishing problem and enhancing long-term temporal modeling performance. Additionally, we propose the Adaptive Skip Recurrent Connection (ASRC), a method that can learn the skip span of skip recurrent connection in each layer of the network. Experiments show that replacing the vanilla recurrent structure in RSNN with SRC significantly improves the model's performance on temporal benchmark datasets. Moreover, ASRC-SNN outperforms SRC-SNN in terms of temporal modeling capabilities and robustness.

Figures (6)

• Figure 1: This figure demonstrates the flow of temporal information within the LIF neurons of the vanilla RSNN, SRC-SNN, and ASRC-SNN models. (a) illustrates that in RSNN, recurrent connections are restricted to adjacent time steps. (b) illustrates that in SRC-SNN, recurrent connections can span multiple time steps. (c) illustrates the dynamic evolution of skip recurrent connections in ASRC-SNN from the start to the end of training. ASRC-SNN initialization: At the beginning of training, each LIF neuron in ASRC-SNN is connected to $T_{\lambda}$ skip recurrent connections, with their weights initialized to $\frac{1}{T_{\lambda}}$. ASRC-SNN during training: During training, the weight distribution of the $T_{\lambda}$ skip recurrent connections becomes more concentrated. ASRC-SNN after training: After training, the weights of the $T_{\lambda}$ skip recurrent connections converge onto a single skip recurrent connection.
• Figure 2: Effects of $\lambda$/$T_{\lambda}$ on SRC-SNN/ASRC-SNN in SSC and PS-MNIST benchmarks. (a)-(b): Impact of $\lambda$ on SRC-SNN; (c)-(d): Impact of $T_{\lambda}$ on ASRC-SNN. All results are shown on PS-MNIST and SSC datasets respectively.
• Figure 3: The impact of the membrane potential decay factor $\alpha$ in LIF neurons on the performance of ASRC-SNN across the PS-MNIST and SSC datasets. (a) and (b) correspond to the results on the PS-MNIST and SSC datasets, respectively.
• Figure 4: The comparison between SRC-SNN and ASRC-SNN under complex datasets and sparse connectivity conditions.
• Figure 5: These plots present heatmaps of the weight distributions of the Softmax kernels across different layers during the training of ASRC-SNN. The x-axis represents the epochs, the y-axis represents time, and each kernel has a size of $T_{\lambda}=51$. (a), (b), and (c) represent the first, second, and third layers, respectively.