Understanding the Convergence in Balanced Resonate-and-Fire Neurons

TL;DR

This work investigates why balanced resonate-and-fire (BRF) neurons enable fast and stable convergence in recurrent spiking neural networks. By introducing three design elements—refractory period, smooth reset, and a divergence boundary—the BRF model achieves a near-identity gradient flow and maintains gradient magnitude through time, backed by a formal analysis of the gradient transition matrix. Empirical results across multiple time-series benchmarks show BRF-RSNNs reach high accuracy quickly with stable training, outperforming or matching ALIF baselines while using fewer spikes. The findings highlight a principled mechanism: constraining the resonator dynamics within a unity-spectral-radius regime yields a smooth, flat error landscape and robust convergence with practical significance for efficient SNN training.

Abstract

Resonate-and-Fire (RF) neurons are an interesting complementary model for integrator neurons in spiking neural networks (SNNs). Due to their resonating membrane dynamics they can extract frequency patterns within the time domain. While established RF variants suffer from intrinsic shortcomings, the recently proposed balanced resonate-and-fire (BRF) neuron marked a significant methodological advance in terms of task performance, spiking and parameter efficiency, as well as, general stability and robustness, demonstrated for recurrent SNNs in various sequence learning tasks. One of the most intriguing result, however, was an immense improvement in training convergence speed and smoothness, overcoming the typical convergence dilemma in backprop-based SNN training. This paper aims at providing further intuitions about how and why these convergence advantages emerge. We show that BRF neurons, in contrast to well-established ALIF neurons, span a very clean and smooth - almost convex - error landscape. Furthermore, empirical results reveal that the convergence benefits are predominantly coupled with a divergence boundary-aware optimization, a major component of the BRF formulation that addresses the numerical stability of the time-discrete resonator approximation. These results are supported by a formal investigation of the membrane dynamics indicating that the gradient is transferred back through time without loss of magnitude.

Figures (6)

• Figure 1: Left: RF neuron behavior when current injected. Divergence (above) and convergence (below) shown with angular frequency of 10 and dampening of -0.3 and -1 higuchi2024. Right: Parameter space of a single BRF neuron with $\delta = 0.01$. Combinations of $\omega$ and $b$ below the divergence boundary leads to convergence.
• Figure 2: Overview over the datasets. Examplary MNIST image and its corresponding sequential and permuted representations. Common pixel row outlined in red on MNIST and S-MNIST sample. ECG sample after level cross encoding. SHD sample after preprocessing. Adapted from higuchi2024.
• Figure 3: S-MNIST, PS-MNIST, ECG, and SHD learning curve between BRF and ALIF model. Each curve averaged per epoch (solid line) with standard deviation (shaded area) over 5 runs. The dot on the curves depict the point at which 95 % of the final accuracy was reached. Adapted from higuchi2024.
• Figure 4: Top: S-MNIST RSNN convergence plot with RF neuon variants. RP: Refractory period; SR: Smooth reset; DB: Divergence boundary. Bottom: S-MNIST (left) and PS-MNIST (right) convergence comparison between (B)RF, standard (non-spiking) LSTM and ALIF model. RF model for PS-MNIST omitted due to loss divergence. Batch size of 256 used for all simulations with 256 hidden units.
• Figure 5: Error landscape plots for RF, BRF, and ALIF network on the S-MNIST dataset. Top: error surface plots. x and y axis correspond to $\alpha$ and $\beta$ and the z axis $f(\alpha, \beta)$. Bottom: the corresponding error contour plots. Note that the value range and hence the coloring does not align across the diagrams for better visualization.