Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A metrological framework for uncertainty evaluation in machine learning classification models

Samuel Bilson, Maurice Cox, Anna Pustogvar, Andrew Thompson

TL;DR

The paper develops a metrological framework for evaluating uncertainty in ML classification outputs by treating the classifier's nominal prediction as a probability mass function over classes. It argues that PMFs, not traditional error metrics alone, are essential for expressing and propagating uncertainty through multistage measurement models, enabling traceability and rigorous uncertainty handling in domains like climate observation and medical diagnostics. The framework analyzes a range of uncertainty statistics (e.g., WVR, UVR, SDM, entropy, IQV, CNV) and demonstrates their behavior on two case studies: land cover classification with Bayesian generative modelling and atrial fibrillation detection with a CNN using Monte Carlo Dropout. Findings indicate entropy is the most sensitive to PMF changes, while UVR provides robust uncertainty assessment; for binary tasks several statistics coincide, simplifying interpretation. The work highlights opportunities to extend the GUM for nominal properties and to better integrate metrology with uncertainty-aware ML pipelines in practice.

Abstract

Machine learning (ML) classification models are increasingly being used in a wide range of applications where it is important that predictions are accompanied by uncertainties, including in climate and earth observation, medical diagnosis and bioaerosol monitoring. The output of an ML classification model is a type of categorical variable known as a nominal property in the International Vocabulary of Metrology (VIM). However, concepts related to uncertainty evaluation for nominal properties are not defined in the VIM, nor is such evaluation addressed by the Guide to the Expression of Uncertainty in Measurement (GUM). In this paper we propose a metrological conceptual uncertainty evaluation framework for nominal properties. This framework is based on probability mass functions and summary statistics thereof, and it is applicable to ML classification. We also illustrate its use in the context of two applications that exemplify the issues and have significant societal impact, namely, climate and earth observation and medical diagnosis. Our framework would enable an extension of the GUM to uncertainty for nominal properties, which would make both applicable to ML classification models.

A metrological framework for uncertainty evaluation in machine learning classification models

TL;DR

The paper develops a metrological framework for evaluating uncertainty in ML classification outputs by treating the classifier's nominal prediction as a probability mass function over classes. It argues that PMFs, not traditional error metrics alone, are essential for expressing and propagating uncertainty through multistage measurement models, enabling traceability and rigorous uncertainty handling in domains like climate observation and medical diagnostics. The framework analyzes a range of uncertainty statistics (e.g., WVR, UVR, SDM, entropy, IQV, CNV) and demonstrates their behavior on two case studies: land cover classification with Bayesian generative modelling and atrial fibrillation detection with a CNN using Monte Carlo Dropout. Findings indicate entropy is the most sensitive to PMF changes, while UVR provides robust uncertainty assessment; for binary tasks several statistics coincide, simplifying interpretation. The work highlights opportunities to extend the GUM for nominal properties and to better integrate metrology with uncertainty-aware ML pipelines in practice.

Abstract

Machine learning (ML) classification models are increasingly being used in a wide range of applications where it is important that predictions are accompanied by uncertainties, including in climate and earth observation, medical diagnosis and bioaerosol monitoring. The output of an ML classification model is a type of categorical variable known as a nominal property in the International Vocabulary of Metrology (VIM). However, concepts related to uncertainty evaluation for nominal properties are not defined in the VIM, nor is such evaluation addressed by the Guide to the Expression of Uncertainty in Measurement (GUM). In this paper we propose a metrological conceptual uncertainty evaluation framework for nominal properties. This framework is based on probability mass functions and summary statistics thereof, and it is applicable to ML classification. We also illustrate its use in the context of two applications that exemplify the issues and have significant societal impact, namely, climate and earth observation and medical diagnosis. Our framework would enable an extension of the GUM to uncertainty for nominal properties, which would make both applicable to ML classification models.

Paper Structure

This paper contains 35 sections, 38 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Machine learning classification from a metrology perspective
  3. Uncertainty evaluation for machine learning classification
  4. The uncertainty evaluation task for machine learning classification models
  5. Related metrological frameworks
  6. Related metrological frameworks
  7. Summary of contributions
  8. Outline of contents
  9. Nominal variables
  10. Nominal versus other variables
  11. Nominal variables in metrology
  12. Uncertainty evaluation for nominal properties
  13. Probability distributions
  14. Expressions of uncertainty
  15. Uncertainty and confidence
  16. ...and 20 more sections

Figures (5)

  • Figure 1: Factors contributing to uncertainty in ML prediction.
  • Figure 2: Cloud masking of Sentinel-2 satellite imagery.
  • Figure 3: Example predictive PMFs as output from ML classifiers.
  • Figure 4: Confusion matrices using BQDA. Final column corresponds to the true/false positive rates for each class. Final row corresponds to true/false negative rates.
  • Figure 5: Confusion matrix for AF classifier on test dataset.