Second-Order Uncertainty Quantification: Variance-Based Measures

TL;DR

This work addresses quantifying second-order uncertainty in probabilistic classification by introducing variance-based measures that decompose uncertainty into label-wise aleatoric ($\mathrm{AU}_k$) and epistemic ($\mathrm{EU}_k$) components using the law of total variance. By defining $\Theta_k=P(Y_k=1)$ and modeling $\Theta=(\Theta_1,...,\Theta_K)$ with a second-order distribution $Q\in\Delta_K^{(2)}$, the authors derive total, aleatoric, and epistemic uncertainties as $\mathrm{TU}_k=\mathrm{Var}(Y_k)$, $\mathrm{AU}_k=\mathbb{E}[\Theta_k(1-\Theta_k)]$, and $\mathrm{EU}_k=\mathrm{Var}(\Theta_k)$, with global versions using weights $w_k$. They argue and prove that these variance-based measures satisfy a set of axioms (A0–A7) ensuring non-negativity, correct limiting behavior, and sensible aggregation across partitions and mean-preserving spreads. Empirically, the variance-based measures are competitive with entropy-based baselines on accuracy-rejection curves, correct/incorrect prediction analyses, and out-of-distribution detection, while offering finer-grained label-wise uncertainty insights beneficial for risk-aware decision making. Overall, the approach provides a principled, interpretable alternative to entropy for second-order uncertainty in classification, with demonstrated practical utility in safety-critical contexts and clear directions for future work on loss-based uncertainty decompositions.

Abstract

Uncertainty quantification is a critical aspect of machine learning models, providing important insights into the reliability of predictions and aiding the decision-making process in real-world applications. This paper proposes a novel way to use variance-based measures to quantify uncertainty on the basis of second-order distributions in classification problems. A distinctive feature of the measures is the ability to reason about uncertainties on a class-based level, which is useful in situations where nuanced decision-making is required. Recalling some properties from the literature, we highlight that the variance-based measures satisfy important (axiomatic) properties. In addition to this axiomatic approach, we present empirical results showing the measures to be effective and competitive to commonly used entropy-based measures.

Figures (10)

• Figure 1: Second-order distributions over the parameter $\theta$ of a (first-order) Bernoulli distribution with the respective uncertainties (entropy-based / variance-based). Note that the variance-based measures are normalized to $[0,1]$.
• Figure 2: Accuracy-rejection curves on CIFAR10 (left), and SVHN (right).
• Figure 3: Histograms of $\text{TU}$ values on CIFAR10 (top), and SVHN (bottom).
• Figure 4: MNIST instances with high uncertainty and their respective label-wise uncertainties.
• Figure 5: Accuracy-rejection curves on MNIST (left), and FMNIST (right).

Theorems & Definitions (8)

• Definition 2.1
• Theorem 3.1
• Lemma 3.1
• Corollary 3.1
• proof : Proof of Theorem \ref{['thm:axioms']}
• proof : Proof of Lemma \ref{['lem:concavity']}
• Proposition A.1
• proof