Random Feature Spiking Neural Networks

TL;DR

The paper tackles the challenge of training Spiking Neural Networks (SNNs) by introducing S-SWIM, a data-driven Random Feature Method that transfers the SWIM paradigm to Spike Response Model (SRM) networks, enabling gradient-free end-to-end training and effective initialization for gradient-based methods. It provides a rigorous mathematical framework and a modular algorithm that samples weights and temporal parameters from data-driven distributions, while solving a linear problem for output weights. The results on time-series forecasting show that S-SWIM achieves high accuracy with substantial speedups over surrogate-gradient training, and ablation studies reinforce the importance of data-driven weight construction and temporal parameter diversification. The approach is interpretable and flexible, but its current limitations include performance on deep networks and the need for robust sampling strategies, suggesting clear avenues for future work and broader applicability.

Abstract

Spiking Neural Networks (SNNs) as Machine Learning (ML) models have recently received a lot of attention as a potentially more energy-efficient alternative to conventional Artificial Neural Networks. The non-differentiability and sparsity of the spiking mechanism can make these models very difficult to train with algorithms based on propagating gradients through the spiking non-linearity. We address this problem by adapting the paradigm of Random Feature Methods (RFMs) from Artificial Neural Networks (ANNs) to Spike Response Model (SRM) SNNs. This approach allows training of SNNs without approximation of the spike function gradient. Concretely, we propose a novel data-driven, fast, high-performance, and interpretable algorithm for end-to-end training of SNNs inspired by the SWIM algorithm for RFM-ANNs, which we coin S-SWIM. We provide a thorough theoretical discussion and supplementary numerical experiments showing that S-SWIM can reach high accuracies on time series forecasting as a standalone strategy and serve as an effective initialisation strategy before gradient-based training. Additional ablation studies show that our proposed method performs better than random sampling of network weights.

Figures (6)

• Figure 1: Overview of the network architecture and the main idea of the training algorithm. (a) The network consists of successive fully connected layers of SRM neurons. Information propagates between layers only as spike trains. Depending on the task, inputs and outputs can be spike trains or real-valued functions. (b) Computation of a single SRM neuron: Incoming spikes are linearly combined and transformed into a continuous potential through convolution with the spike response kernel (SRK) $k$ . After shifting by the bias $b$, outgoing spikes are generated through thresholding. Refractory contributions are generated by convolving the outgoing spikes with the refractory kernel (RfK) $q$ and weighting by the spike-cost $c$. Temporal parameters are not shown for simplicity. (cf. definition \ref{['def:ff_snn']}). (c) The weights of the hidden layers are chosen to separate the membrane potentials generated by samples with similar inputs and dissimilar targets (cf. sections \ref{['sec:sswim_h_dist']} and \ref{['sec:sswim_h_weights']}). The weights of the final layer are found by solving a linear problem (cf. section \ref{['sec:sswim_o_fit']}).
• Figure 2: Training time of S-SWIM and SGD training across datasets and prediction horizons $H$ for the conducted forecasting experiments with the $\text{Hat}$ kernel. Note that the time is given in $\log$ scale.
• Figure 3: Classification accuracy across different architectures and configurations (Number of hidden layers, number of neurons, weight constructions and sampling metrics (cf. section \ref{['app:sswim_h_dist']})).
• Figure 4: Classification accuracy using spiking readout.
• Figure 5: Results of the ablation study. Arrows indicate significantly underperforming models. Missing results indicate errors during the computation.

Theorems & Definitions (7)

• Definition 3.1
• Remark 3.2
• Proposition A.1
• proof
• Proposition A.2
• proof
• Corollary A.3