Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning

Zachary Nado, Neil Band, Mark Collier, Josip Djolonga, Michael W. Dusenberry, Sebastian Farquhar, Qixuan Feng, Angelos Filos, Marton Havasi, Rodolphe Jenatton, Ghassen Jerfel, Jeremiah Liu, Zelda Mariet, Jeremy Nixon, Shreyas Padhy, Jie Ren, Tim G. J. Rudner, Faris Sbahi, Yeming Wen, Florian Wenzel, Kevin Murphy, D. Sculley, Balaji Lakshminarayanan, Jasper Snoek, Yarin Gal, Dustin Tran

TL;DR

This paper tackles the challenge of benchmarking uncertainty and robustness in deep learning by introducing Uncertainty Baselines, a comprehensive, forkable library of high-quality baselines. It provides 83 baselines spanning 19 methods across 9 tasks, with standardized pipelines, multiple evaluation metrics, and reproducibility artifacts such as model checkpoints and TensorBoard dashboards. The framework is modular, framework-agnostic, and designed for easy extension, with careful attention to preprocessing, model choices, hyperparameter tuning, and open data. The work enables reliable, scalable comparisons and practical reuse for research and applied settings, particularly in calibration, selective prediction, and OOD robustness.

Abstract

High-quality estimates of uncertainty and robustness are crucial for numerous real-world applications, especially for deep learning which underlies many deployed ML systems. The ability to compare techniques for improving these estimates is therefore very important for research and practice alike. Yet, competitive comparisons of methods are often lacking due to a range of reasons, including: compute availability for extensive tuning, incorporation of sufficiently many baselines, and concrete documentation for reproducibility. In this paper we introduce Uncertainty Baselines: high-quality implementations of standard and state-of-the-art deep learning methods on a variety of tasks. As of this writing, the collection spans 19 methods across 9 tasks, each with at least 5 metrics. Each baseline is a self-contained experiment pipeline with easily reusable and extendable components. Our goal is to provide immediate starting points for experimentation with new methods or applications. Additionally we provide model checkpoints, experiment outputs as Python notebooks, and leaderboards for comparing results. Code available at https://github.com/google/uncertainty-baselines.

Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning

TL;DR

This paper tackles the challenge of benchmarking uncertainty and robustness in deep learning by introducing Uncertainty Baselines, a comprehensive, forkable library of high-quality baselines. It provides 83 baselines spanning 19 methods across 9 tasks, with standardized pipelines, multiple evaluation metrics, and reproducibility artifacts such as model checkpoints and TensorBoard dashboards. The framework is modular, framework-agnostic, and designed for easy extension, with careful attention to preprocessing, model choices, hyperparameter tuning, and open data. The work enables reliable, scalable comparisons and practical reuse for research and applied settings, particularly in calibration, selective prediction, and OOD robustness.

Abstract

High-quality estimates of uncertainty and robustness are crucial for numerous real-world applications, especially for deep learning which underlies many deployed ML systems. The ability to compare techniques for improving these estimates is therefore very important for research and practice alike. Yet, competitive comparisons of methods are often lacking due to a range of reasons, including: compute availability for extensive tuning, incorporation of sufficiently many baselines, and concrete documentation for reproducibility. In this paper we introduce Uncertainty Baselines: high-quality implementations of standard and state-of-the-art deep learning methods on a variety of tasks. As of this writing, the collection spans 19 methods across 9 tasks, each with at least 5 metrics. Each baseline is a self-contained experiment pipeline with easily reusable and extendable components. Our goal is to provide immediate starting points for experimentation with new methods or applications. Additionally we provide model checkpoints, experiment outputs as Python notebooks, and leaderboards for comparing results. Code available at https://github.com/google/uncertainty-baselines.

Paper Structure

This paper contains 9 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Uncertainty Baselines
  3. Results
  4. Dataset Details
  5. Model Details
  6. Hyperparameter Tuning
  7. Diabetic Retinopathy benchmark tuning details.
  8. Supported Baselines
  9. Open-Source Data

Figures (4)

  • Figure 1: The structure for an experiment under the TensorFlow or Pytorch backend. One instantiates a dataset (Cifar10Dataset or DiabeticRetinopathyDataset) and model (wide_resnet or resnet50_torch) within an end-to-end training script. After training, one inputs saved model checkpoints into Robustness Metrics for evaluation.
  • Figure 2: Performance analysis of a MIMO baseline on a TPUv3-32 using the https://www.tensorflow.org/tensorboard/tensorboard_profiling_keras. The runtime is optimized, bound only by model operations, an irreducible bottleneck for a given baseline. Our implementations have 100% utilization of the TPU devices.
  • Figure 3: 8 baselines evaluated on ImageNet, ImageNet-C, ImageNet-A, and ImageNetV2 (matched frequency variant). (top) Top-1 accuracy. (bottom) Expected calibration error. Results demonstrate the many baselines available with competitive performance.
  • Figure 4: ImageNet baseline applied to deferred prediction. In this task, one defers predictions according to the model's confidence (left) or a desired data retention rate (right).