Average-Over-Time Spiking Neural Networks for Uncertainty Estimation in Regression

TL;DR

This work tackles uncertainty estimation in regression for spiking neural networks (SNNs) by adapting the Average-Over-Time SNN (AOT-SNN) framework to two probabilistic paradigms: (i) a heteroscedastic Gaussian model producing a time-averaged mean $\\mu_*(\boldsymbol{x})$ and variance $\\sigma_*^2(\boldsymbol{x})$, and (ii) a Regression-as-Classification (RAC) formulation discretizing the target into $K$ bins with class probabilities $p_k$. The RAC path uses a distance-based loss combining a distance penalization and entropy regularization to yield well-calibrated predictive distributions, while the Gaussian path relies on time-step outputs $ (\\mu_t, \\sigma_t^2)$ aggregated over $T$ steps. Across a toy dataset and multiple UCI benchmarks, RAC-AOT-SNN consistently delivers strong RMSE and competitive or superior NLL compared to deep neural network baselines, with Gaussian-AOT-SNN also performing well, particularly in RMSE. The results demonstrate that energy-efficient, time-averaged SNNs can provide high-quality uncertainty estimates for regression tasks, offering a practical alternative for real-time, resource-constrained deployments.

Abstract

Uncertainty estimation is a standard tool to quantify the reliability of modern deep learning models, and crucial for many real-world applications. However, efficient uncertainty estimation methods for spiking neural networks, particularly for regression models, have been lacking. Here, we introduce two methods that adapt the Average-Over-Time Spiking Neural Network (AOT-SNN) framework to regression tasks, enhancing uncertainty estimation in event-driven models. The first method uses the heteroscedastic Gaussian approach, where SNNs predict both the mean and variance at each time step, thereby generating a conditional probability distribution of the target variable. The second method leverages the Regression-as-Classification (RAC) approach, reformulating regression as a classification problem to facilitate uncertainty estimation. We evaluate our approaches on both a toy dataset and several benchmark datasets, demonstrating that the proposed AOT-SNN models achieve performance comparable to or better than state-of-the-art deep neural network methods, particularly in uncertainty estimation. Our findings highlight the potential of SNNs for uncertainty estimation in regression tasks, providing an efficient and biologically inspired alternative for applications requiring both accuracy and energy efficiency.

Figures (3)

• Figure 1: a, The MC-dropout method estimates uncertainty by averaging the outputs of multiple forward passes through a dropout-enabled DNN. b, Deep ensembles train an ensemble of DNNs with identical architectures. For regression tasks, each member in an ensemble assumes a heteroscedastic Gaussian distribution and outputs two variables per input: the mean $\mu_m(\mathbf{x})$ and variance $\sigma_m^2(\mathbf{x})$. The predictive mean $\mu_*(\mathbf{x})$ and variance $\sigma_*(\mathbf{x})^2$ of the ensemble are derived from these two variables. c, In classification, AOT-SNNs apply continual MC-dropout in SNNs by averaging the probabilities across time steps to generate the predictive distribution. d, In regression, at each time step, an AOT-SNN outputs the the mean $\mu_t(\mathbf{x})$ and variance $\sigma_t^2(\mathbf{x})$, with the final predictive mean $\mu_*(\mathbf{x})$ and variance $\sigma_*(\mathbf{x})^2$ calculated from these values.
• Figure 2: Illustration of the heteroscedastic Gaussian and Regression-as-Classification (RAC) approaches for uncertainty estimation in regression tasks. (a) Neural network architecture for predicting mean and variance. (b) Data fitting with the heteroscedastic Gaussian assumption. (c) RAC approach, discretizing the target variable into bins and predicting conditional probabilities. (d) Data fitting with the RAC approach under the uniform distribution assumption.
• Figure 3: Comparison of RMSE and NLL metrics for AOT-SNN and DNN models using Gaussian and RAC approaches on the toy dataset. The Gaussian-based AOT-SNN model shows improved RMSE and NLL compared to its DNN counterpart, while the RAC-based AOT-SNN achieves comparable results to the RAC DNN model.