Surrogates, Spikes, and Sparsity: Performance Analysis and Characterization of SNN Hyperparameters on Hardware

Abstract

Spiking Neural Networks (SNNs) offer inherent advantages for low-power inference through sparse, event-driven computation. However, the theoretical energy benefits of SNNs are often decoupled from real hardware performance due to the opaque relationship between training-time choices and inference-time sparsity. While prior work has focused on weight pruning and compression, the role of training hyperparameters -- specifically surrogate gradient functions and neuron model configurations -- in shaping hardware-level activation sparsity remains underexplored. This paper presents a workload characterization study quantifying the sensitivity of hardware latency to SNN hyperparameters. We decouple the impact of surrogate gradient functions (e.g., Fast Sigmoid, Spike Rate Escape) and neuron models (LIF, Lapicque) on classification accuracy and inference efficiency across three event-based vision datasets: DVS128-Gesture, N-MNIST, and DVS-CIFAR10. Our analysis reveals that standard accuracy metrics are poor predictors of hardware efficiency. While Fast Sigmoid achieves the highest accuracy on DVS-CIFAR10, Spike Rate Escape reduces inference latency by up to 12.2% on DVS128-Gesture with minimal accuracy trade-offs. We also demonstrate that neuron model selection is as critical as parameter tuning; transitioning from LIF to Lapicque neurons yields up to 28% latency reduction. We validate on a custom cycle-accurate FPGA-based SNN instrumentation platform, showing that sparsity-aware hyperparameter selection can improve accuracy by 9.1% and latency by over 2x compared to baselines. These findings establish a methodology for predicting hardware behavior from training parameters. The RTL and reproducibility artifacts are at https://zenodo.org/records/18893738.

Figures (7)

• Figure 1: Hardware dataflow comparison between the LIF and Lapicque neuron models. LIF uses bit-shifts for decay (3 ops/update); Lapicque requires explicit multiplications for RC constants (4 ops + 2$\times$ MUL). Despite higher per-update cost, Lapicque's temporal dynamics suppress total spike events, yielding lower system-level latency.
• Figure 2: Visualization of the derivative profiles ($\frac{\partial S}{\partial U}$) for different surrogate functions plotted against the membrane potential's proximity to the firing threshold ($U - U_{thr}$). Fast Sigmoid (a) maintains meaningful gradient support in the tails as $k$ increases. Arctangent (b) exhibits similar behavior with Cauchy-like tails that preserve gradient flow farther from the threshold. Spike Rate Escape (c) exhibits exponential decay, leading to vanishing gradients (gradient $\approx$ 0) for neurons far from the threshold (assuming $\beta = k$). The Stochastic Spike Operator (d) uses a constant gradient of 1 above threshold and injects uniform noise below threshold, enabling gradient flow through sub-threshold neurons. These mathematical properties explain the sharp accuracy degradation ("cliffs") observed in our experimental characterization (Section \ref{['sec:results']}).
• Figure 3: Overview of our design space exploration workflow.
• Figure 4: Dataflow of the hardware instrumentation platform (Parametric Sparsity-Aware Convolution Engine). The design features an Event Control Unit (ECU) that dynamically schedules only active neurons, and Neural Cores (NCs) with configurable accumulation logic. This event-driven architecture allows for precise isolation of latency reductions caused by algorithmic sparsity tuning.
• Figure 5: Accuracy trends for surrogate gradient functions across three datasets. Each plot shows classification accuracy as a function of the derivative scaling factor (slope) $\alpha$ for four surrogate functions: Fast Sigmoid, Arctangent, SRE, and SSO. For each surrogate, representative top-performing $\alpha$ values were selected from 40 Optuna trials per candidate configuration (each trained for up to 200 epochs with early stopping).