Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Survey on Uncertainty Quantification Methods for Deep Learning

Wenchong He, Zhe Jiang, Tingsong Xiao, Zelin Xu, Yukun Li

TL;DR

<3-5 sentence high-level summary> This survey advances uncertainty quantification for deep learning by reframing the problem around uncertainty sources rather than solely network architectures or Bayesian formalisms. It surveys model, data, and combined approaches, detailing Bayesian neural networks, ensembles, Gaussian processes, discriminative and generative models, evidential learning, and conformal prediction, and it links these methods to OOD detection, active learning, and reinforcement learning. The paper also outlines future directions, including UQ for large language models, scientific simulations, and structured outputs, and stresses the need for efficient, calibrated, and explainable uncertainty estimates. Its taxonomy and critical comparisons aim to guide practitioners in selecting appropriate UQ methods for real-world, high-stakes applications.</3-5 sentence high-level summary>

Abstract

Deep neural networks (DNNs) have achieved tremendous success in computer vision, natural language processing, and scientific and engineering domains. However, DNNs can make unexpected, incorrect, yet overconfident predictions, leading to serious consequences in high-stakes applications such as autonomous driving, medical diagnosis, and disaster response. Uncertainty quantification (UQ) estimates the confidence of DNN predictions in addition to their accuracy. In recent years, many UQ methods have been developed for DNNs. It is valuable to systematically categorize these methods and compare their strengths and limitations. Existing surveys mostly categorize UQ methodologies by neural network architecture or Bayesian formulation, while overlooking the uncertainty sources each method addresses, making it difficult to select an appropriate approach in practice. To fill this gap, this paper presents a taxonomy of UQ methods for DNNs based on uncertainty sources (e.g., data versus model uncertainty). We summarize the advantages and disadvantages of each category, and illustrate how UQ can be applied to machine learning problems (e.g., active learning, out-of-distribution robustness, and deep reinforcement learning). We also identify future research directions, including UQ for large language models (LLMs), AI-driven scientific simulations, and deep neural networks with structured outputs.

A Survey on Uncertainty Quantification Methods for Deep Learning

TL;DR

<3-5 sentence high-level summary> This survey advances uncertainty quantification for deep learning by reframing the problem around uncertainty sources rather than solely network architectures or Bayesian formalisms. It surveys model, data, and combined approaches, detailing Bayesian neural networks, ensembles, Gaussian processes, discriminative and generative models, evidential learning, and conformal prediction, and it links these methods to OOD detection, active learning, and reinforcement learning. The paper also outlines future directions, including UQ for large language models, scientific simulations, and structured outputs, and stresses the need for efficient, calibrated, and explainable uncertainty estimates. Its taxonomy and critical comparisons aim to guide practitioners in selecting appropriate UQ methods for real-world, high-stakes applications.</3-5 sentence high-level summary>

Abstract

Deep neural networks (DNNs) have achieved tremendous success in computer vision, natural language processing, and scientific and engineering domains. However, DNNs can make unexpected, incorrect, yet overconfident predictions, leading to serious consequences in high-stakes applications such as autonomous driving, medical diagnosis, and disaster response. Uncertainty quantification (UQ) estimates the confidence of DNN predictions in addition to their accuracy. In recent years, many UQ methods have been developed for DNNs. It is valuable to systematically categorize these methods and compare their strengths and limitations. Existing surveys mostly categorize UQ methodologies by neural network architecture or Bayesian formulation, while overlooking the uncertainty sources each method addresses, making it difficult to select an appropriate approach in practice. To fill this gap, this paper presents a taxonomy of UQ methods for DNNs based on uncertainty sources (e.g., data versus model uncertainty). We summarize the advantages and disadvantages of each category, and illustrate how UQ can be applied to machine learning problems (e.g., active learning, out-of-distribution robustness, and deep reinforcement learning). We also identify future research directions, including UQ for large language models (LLMs), AI-driven scientific simulations, and deep neural networks with structured outputs.
Paper Structure (36 sections, 25 equations, 13 figures, 5 tables)

This paper contains 36 sections, 25 equations, 13 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Types of uncertainty source
  3. Preliminaries of Supervised Learning
  4. Model uncertainty
  5. Sources of model uncertainty
  6. Model uncertainty representation
  7. Data uncertainty
  8. Source of data uncertainty
  9. Data uncertainty representation
  10. Application domains
  11. A Taxonomy of UQ Methodologies for DNNs
  12. Model Uncertainty
  13. Bayesian Neural Networks
  14. Ensemble models
  15. Sample distribution-related methods
  16. ...and 21 more sections

Figures (13)

  • Figure 1: Existing survey on UQ methods for DNNs.
  • Figure 2: Visualization on various model uncertainty sources.
  • Figure 3: Data uncertainty visualization examples (Different colors represent samples in different classes).
  • Figure 4: Different types of uncertainty source.
  • Figure 5: A taxonomy for existing literature on UQ for DNN.
  • ...and 8 more figures