Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robustness quantification: a new method for assessing the reliability of the predictions of a classifier

Adrián Detavernier, Jasper De Bock

TL;DR

The paper tackles the challenge of per-instance reliability for generative probabilistic classifiers under data scarcity and distribution shift. It introduces robustness quantification, an instance-specific reliability metric based on minimal perturbations of the joint distribution, and contrasts it with uncertainty quantification. The authors develop global and local perturbation schemes, derive closed-form robustness metrics for Naive Bayes, and demonstrate through synthetic experiments that robustness remains stable under adverse conditions while uncertainty degrades. They conclude robustness quantification provides a practical reliability signal and suggest combining it with uncertainty metrics and extending the framework to more complex models.

Abstract

Based on existing ideas in the field of imprecise probabilities, we present a new approach for assessing the reliability of the individual predictions of a generative probabilistic classifier. We call this approach robustness quantification, compare it to uncertainty quantification, and demonstrate that it continues to work well even for classifiers that are learned from small training sets that are sampled from a shifted distribution.

Robustness quantification: a new method for assessing the reliability of the predictions of a classifier

TL;DR

The paper tackles the challenge of per-instance reliability for generative probabilistic classifiers under data scarcity and distribution shift. It introduces robustness quantification, an instance-specific reliability metric based on minimal perturbations of the joint distribution, and contrasts it with uncertainty quantification. The authors develop global and local perturbation schemes, derive closed-form robustness metrics for Naive Bayes, and demonstrate through synthetic experiments that robustness remains stable under adverse conditions while uncertainty degrades. They conclude robustness quantification provides a practical reliability signal and suggest combining it with uncertainty metrics and extending the framework to more complex models.

Abstract

Based on existing ideas in the field of imprecise probabilities, we present a new approach for assessing the reliability of the individual predictions of a generative probabilistic classifier. We call this approach robustness quantification, compare it to uncertainty quantification, and demonstrate that it continues to work well even for classifiers that are learned from small training sets that are sampled from a shifted distribution.

Paper Structure

This paper contains 30 sections, 3 theorems, 25 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Classification
  3. Classifiers
  4. Generative probabilistic classifiers
  5. Learning a classifier
  6. Reliability quantification
  7. Uncertainty
  8. Robustness
  9. Uncertainty quantification
  10. Maximum probability
  11. Entropy
  12. Aleatoric uncertainty
  13. Total uncertainty
  14. Epistemic uncertainty
  15. Robustness Quantification
  16. ...and 15 more sections

Key Result

theorem 1

Let $h_{p}$ be a generative classifier corresponding to a mass function $p$. Let $\mathcal{P}$ be a perturbation of $p$ and let $\hat{c}$ be the prediction according to $h_{p}$ for the set of features $f$. Then $\hat{c}$ is robust w.r.t. the perturbation $\mathcal{P}$ if and only if where the first inequality should be checked first because if it fails, the fraction in the second inequality is un

Figures (2)

  • Figure 1: The means of the accuracy-acceptance curves for decreasing $N_{\mathrm{train}}$ (100, 50, 35 from top to bottom) and increasing $\gamma$ (0, 0.2, 0.4 left to right).
  • Figure 2: The standard deviations of the accuracy-acceptance curves for decreasing $N_{\mathrm{train}}$ (100, 50, 25 from top to bottom) and increasing $\gamma$ (0, 0.2, 0.4 left to right).

Theorems & Definitions (11)

  • definition 1
  • definition 2
  • theorem 1
  • proof
  • definition 3
  • definition 4
  • definition 5
  • theorem 2
  • proof
  • theorem 3
  • ...and 1 more