Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uncertainty Quantification With Noise Injection in Neural Networks: A Bayesian Perspective

Xueqiong Yuan, Jipeng Li, Ercan Engin Kuruoglu

TL;DR

This work tackles model uncertainty in neural networks by establishing that injecting noise into weights is a Bayesian approximation of a deep Gaussian process ($\mathcal{GP}$) in the infinite-width limit. It then introduces Monte Carlo Noise Injection (MCNI), a scalable method that trains with weight noise and uses multiple forward passes at inference to estimate predictive uncertainty via a Bayesian posterior. The authors prove an ELBO-equivalence to a GP-based objective and demonstrate improved predictive accuracy, calibration, and selective performance on toy data, UCI regression tasks, and CIFAR-10 compared with MC dropout. Overall, MCNI provides a principled, practical approach to uncertainty quantification in deep networks with competitive performance across regression and classification benchmarks.

Abstract

Model uncertainty quantification involves measuring and evaluating the uncertainty linked to a model's predictions, helping assess their reliability and confidence. Noise injection is a technique used to enhance the robustness of neural networks by introducing randomness. In this paper, we establish a connection between noise injection and uncertainty quantification from a Bayesian standpoint. We theoretically demonstrate that injecting noise into the weights of a neural network is equivalent to Bayesian inference on a deep Gaussian process. Consequently, we introduce a Monte Carlo Noise Injection (MCNI) method, which involves injecting noise into the parameters during training and performing multiple forward propagations during inference to estimate the uncertainty of the prediction. Through simulation and experiments on regression and classification tasks, our method demonstrates superior performance compared to the baseline model.

Uncertainty Quantification With Noise Injection in Neural Networks: A Bayesian Perspective

TL;DR

This work tackles model uncertainty in neural networks by establishing that injecting noise into weights is a Bayesian approximation of a deep Gaussian process () in the infinite-width limit. It then introduces Monte Carlo Noise Injection (MCNI), a scalable method that trains with weight noise and uses multiple forward passes at inference to estimate predictive uncertainty via a Bayesian posterior. The authors prove an ELBO-equivalence to a GP-based objective and demonstrate improved predictive accuracy, calibration, and selective performance on toy data, UCI regression tasks, and CIFAR-10 compared with MC dropout. Overall, MCNI provides a principled, practical approach to uncertainty quantification in deep networks with competitive performance across regression and classification benchmarks.

Abstract

Model uncertainty quantification involves measuring and evaluating the uncertainty linked to a model's predictions, helping assess their reliability and confidence. Noise injection is a technique used to enhance the robustness of neural networks by introducing randomness. In this paper, we establish a connection between noise injection and uncertainty quantification from a Bayesian standpoint. We theoretically demonstrate that injecting noise into the weights of a neural network is equivalent to Bayesian inference on a deep Gaussian process. Consequently, we introduce a Monte Carlo Noise Injection (MCNI) method, which involves injecting noise into the parameters during training and performing multiple forward propagations during inference to estimate the uncertainty of the prediction. Through simulation and experiments on regression and classification tasks, our method demonstrates superior performance compared to the baseline model.
Paper Structure (21 sections, 1 theorem, 14 equations, 3 figures, 4 tables)

This paper contains 21 sections, 1 theorem, 14 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Weight Noise Injection
  4. Model Uncertainty Quantification
  5. THEORETICAL PROOF
  6. Background Material
  7. Gaussian Processes
  8. Variational Inference
  9. Weight Noise Injection As a Bayesian Approximation
  10. Experiments on Toy Dataset
  11. Experiment Configuration
  12. Metrics
  13. Result
  14. Experiments on Real-World Datasets
  15. Datasets and Models
  16. ...and 6 more sections

Key Result

Theorem 1

A multi-layer fully-connected BNN with an i.i.d. prior over its parameters is equivalent to a Gaussian process (GP), in the limit of infinite network width. Specifically, let $x$ be the input of the neural network, $b$ be the bias term, $W_{ij}^l$ be the element at position $(i,j)$ of the weight mat where $y_j^l(x)$ are i.i.d. for every $j$. Then, $z_i^l \sim \mathcal{GP}(0, K^l)$, where $K^l$ is

Figures (3)

  • Figure 1: Predictions made by each method on the toy dataset. The data points are represented by light blue points, the true data generating function $y=0.3\sin \pi x$ is depicted as the dark blue line and the average predictions are shown as a black line. Shaded areas represent regions of predicted means $\pm 3$ standard deviations.
  • Figure 2: Softmax outputs from the deterministic NN, MCNI, and MC dropout. The x-axis represents the standard deviation of the Gaussian noise added to the data.
  • Figure 3: Risk-Coverage curve on datasets Boston Housing, Kin8nm, and CIFAR10. The risk of the regression task is measured by RMSE and the risk of the classification task is measured by the accuracy.

Theorems & Definitions (1)

  • Theorem 1