Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lightning UQ Box: A Comprehensive Framework for Uncertainty Quantification in Deep Learning

Nils Lehmann, Jakob Gawlikowski, Adam J. Stewart, Vytautas Jancauskas, Stefan Depeweg, Eric Nalisnick, Nina Maria Gottschling

TL;DR

A theoretical and quantitative comparison of the wide range of state-of-the-art UQ methods implemented in theLightning UQ Box, a unified interface for applying and evaluating various approaches to UQ.

Abstract

Uncertainty quantification (UQ) is an essential tool for applying deep neural networks (DNNs) to real world tasks, as it attaches a degree of confidence to DNN outputs. However, despite its benefits, UQ is often left out of the standard DNN workflow due to the additional technical knowledge required to apply and evaluate existing UQ procedures. Hence there is a need for a comprehensive toolbox that allows the user to integrate UQ into their modelling workflow, without significant overhead. We introduce \texttt{Lightning UQ Box}: a unified interface for applying and evaluating various approaches to UQ. In this paper, we provide a theoretical and quantitative comparison of the wide range of state-of-the-art UQ methods implemented in our toolbox. We focus on two challenging vision tasks: (i) estimating tropical cyclone wind speeds from infrared satellite imagery and (ii) estimating the power output of solar panels from RGB images of the sky. By highlighting the differences between methods our results demonstrate the need for a broad and approachable experimental framework for UQ, that can be used for benchmarking UQ methods. The toolbox, example implementations, and further information are available at: https://github.com/lightning-uq-box/lightning-uq-box

Lightning UQ Box: A Comprehensive Framework for Uncertainty Quantification in Deep Learning

TL;DR

A theoretical and quantitative comparison of the wide range of state-of-the-art UQ methods implemented in theLightning UQ Box, a unified interface for applying and evaluating various approaches to UQ.

Abstract

Uncertainty quantification (UQ) is an essential tool for applying deep neural networks (DNNs) to real world tasks, as it attaches a degree of confidence to DNN outputs. However, despite its benefits, UQ is often left out of the standard DNN workflow due to the additional technical knowledge required to apply and evaluate existing UQ procedures. Hence there is a need for a comprehensive toolbox that allows the user to integrate UQ into their modelling workflow, without significant overhead. We introduce \texttt{Lightning UQ Box}: a unified interface for applying and evaluating various approaches to UQ. In this paper, we provide a theoretical and quantitative comparison of the wide range of state-of-the-art UQ methods implemented in our toolbox. We focus on two challenging vision tasks: (i) estimating tropical cyclone wind speeds from infrared satellite imagery and (ii) estimating the power output of solar panels from RGB images of the sky. By highlighting the differences between methods our results demonstrate the need for a broad and approachable experimental framework for UQ, that can be used for benchmarking UQ methods. The toolbox, example implementations, and further information are available at: https://github.com/lightning-uq-box/lightning-uq-box
Paper Structure (12 sections, 8 figures, 3 tables)

This paper contains 12 sections, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Benchmarking UQ Methods: the Lightning UQ Box
  4. Provided Types of UQ Methods
  5. Experimental Setup for Validation
  6. Datasets
  7. Methodological Setup
  8. Results: Examples of UQ Method Analysis
  9. Selective Prediction for Wind Speed Estimation
  10. Photovoltaic Power Output Estimation Under Cloudy and Sunny Conditions
  11. Conclusion
  12. Ethics and Broader Impact Statement

Figures (8)

  • Figure 1: The structure of Lightning UQ Box. The experiments can be built and evaluated at scale or manually tailored to specific use cases. For large experiments at scale, only a dataset and a configuration file have to be provided.
  • Figure 2: Example code and visualization on toy regression dataset.
  • Figure 3: Visualization of the Tropical Cyclon (left) and the Digital Typhoon Dataset (right).
  • Figure 4: Visualization of SKIPP'D Dataset.
  • Figure 5: Selective Prediction RMSE improvement per category on the Digital Typhoon Dataset (left) and Tropical Cyclone Dataset (right).
  • ...and 3 more figures