On weight and variance uncertainty in neural networks for regression tasks
Moein Monemi, Morteza Amini, S. Mahmoud Taheri, Mohammad Arashi
TL;DR
The paper investigates weight and variance uncertainty in Bayesian neural networks for regression by extending the Bayes by Backprop framework to include a posterior over the likelihood variance. The proposed Variance Uncertainty VB Regression Network (VBNET-SVAR) generalizes the fixed-variance model (VBNET-FIXED) by placing a variational distribution over both weights and the variance parameter, demonstrating improved predictive accuracy and coverage on nonlinear function approximation and high-dimensional riboflavin data. Through experiments, the authors show that modeling variance uncertainty yields better MSPE and more reliable predictive intervals, supporting the practical value of incorporating posterior variance in regression tasks. The work also compares different priors (Gaussian and spike-and-slab) and confirms that variance uncertainty enhances generalization in Bayesian neural networks for regression.
Abstract
We consider the problem of weight uncertainty proposed by [Blundell et al. (2015). Weight uncertainty in neural network. In International conference on machine learning, 1613-1622, PMLR.] in neural networks {(NNs)} specialized for regression tasks. {We further} investigate the effect of variance uncertainty in {their model}. We show that including the variance uncertainty can improve the prediction performance of the Bayesian {NN}. Variance uncertainty enhances the generalization of the model {by} considering the posterior distribution over the variance parameter. { We examine the generalization ability of the proposed model using a function approximation} example and {further illustrate it with} the riboflavin genetic data set. {We explore fully connected dense networks and dropout NNs with} Gaussian and spike-and-slab priors, respectively, for the network weights.