Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Density Uncertainty Layers for Reliable Uncertainty Estimation

Yookoon Park, David M. Blei

TL;DR

A novel criterion for reliable predictive uncertainty is proposed: a model's predictive variance should be grounded in the empirical density of the input, and a stochastic neural network architecture is developed that satisfies the density uncertain criterion by design.

Abstract

Assessing the predictive uncertainty of deep neural networks is crucial for safety-related applications of deep learning. Although Bayesian deep learning offers a principled framework for estimating model uncertainty, the common approaches that approximate the parameter posterior often fail to deliver reliable estimates of predictive uncertainty. In this paper, we propose a novel criterion for reliable predictive uncertainty: a model's predictive variance should be grounded in the empirical density of the input. That is, the model should produce higher uncertainty for inputs that are improbable in the training data and lower uncertainty for inputs that are more probable. To operationalize this criterion, we develop the density uncertainty layer, a stochastic neural network architecture that satisfies the density uncertain criterion by design. We study density uncertainty layers on the UCI and CIFAR-10/100 uncertainty benchmarks. Compared to existing approaches, density uncertainty layers provide more reliable uncertainty estimates and robust out-of-distribution detection performance.

Density Uncertainty Layers for Reliable Uncertainty Estimation

TL;DR

A novel criterion for reliable predictive uncertainty is proposed: a model's predictive variance should be grounded in the empirical density of the input, and a stochastic neural network architecture is developed that satisfies the density uncertain criterion by design.

Abstract

Assessing the predictive uncertainty of deep neural networks is crucial for safety-related applications of deep learning. Although Bayesian deep learning offers a principled framework for estimating model uncertainty, the common approaches that approximate the parameter posterior often fail to deliver reliable estimates of predictive uncertainty. In this paper, we propose a novel criterion for reliable predictive uncertainty: a model's predictive variance should be grounded in the empirical density of the input. That is, the model should produce higher uncertainty for inputs that are improbable in the training data and lower uncertainty for inputs that are more probable. To operationalize this criterion, we develop the density uncertainty layer, a stochastic neural network architecture that satisfies the density uncertain criterion by design. We study density uncertainty layers on the UCI and CIFAR-10/100 uncertainty benchmarks. Compared to existing approaches, density uncertainty layers provide more reliable uncertainty estimates and robust out-of-distribution detection performance.
Paper Structure (19 sections, 1 theorem, 37 equations, 1 figure, 9 tables)

This paper contains 19 sections, 1 theorem, 37 equations, 1 figure, 9 tables.

Table of Contents

  1. Introduction
  2. Density Uncertainty Layers
  3. Motivation: Bayesian Linear Regression
  4. Variational Inference for BNNs
  5. The Density Uncertainty Criterion
  6. The Density Uncertainty Layer
  7. Gaussian energy
  8. Mixture of rank-1 Gaussians
  9. Related Work
  10. Bayesian Neural Networks and Uncertainty
  11. Uncertainty Estimation for Deep Learning
  12. Empirical Studies
  13. Toy Regression
  14. CIFAR-10 and CIFAR-100
  15. Out-of-Distribution Detection on SVHN
  16. ...and 4 more sections

Key Result

Proposition 1

Define the total energy $E(x, h_1, ..., h_\ell) = \sum_{\ell=1}^L E_\ell(h_\ell)$. Assume the followings: Then the dimension-wise variance of the network output $a_{L+1}$ is bounded from below by the expected total energy: for some constant $C$. Proof is in the appendix.

Figures (1)

  • Figure 1: Predictive uncertainty of a two-layer MLP on a toy regression problem. The red dots denote the training data and the blue shades mark the 95th and 99th percentiles of the predictive variance. All baselines other than Density Uncertainty fail to account for the in-between uncertainty in the low-density region around the origin

Theorems & Definitions (2)

  • Definition 1: Density uncertainty criterion
  • Proposition 1: Density Uncertainty Criterion for Residual Networks