Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Integral Imprecise Probability Metrics

Siu Lun Chau, Michele Caprio, Krikamol Muandet

TL;DR

This work introduces Integral Imprecise Probability Metrics (IIPMs), a Choquet-integral generalisation of Integral Probability Metrics (IPMs) to the broad class of imprecise probabilities represented by capacities. It develops Maximum Mean Imputation (MMI), a new epistemic uncertainty (EU) measure derived from distances between a capacity and its conjugate, and demonstrates that IIPMs metrically capture Choquet weak convergence under suitable function classes. Theoretical results include metric properties of IIPM, and practical instantiations such as the Lower Dudley metric and Lower Total Variation, along with connections to the epsilon-contamination Kantorovich problem. Empirically, MMI and its linear-time upper bound (MMI-Lin) outperform traditional EU measures in selective classification across multiclass settings, illustrating both the theory and applicability of IPML for robust uncertainty quantification under imprecision.

Abstract

Quantifying differences between probability distributions is fundamental to statistics and machine learning, primarily for comparing statistical uncertainty. In contrast, epistemic uncertainty (EU) -- due to incomplete knowledge -- requires richer representations than those offered by classical probability. Imprecise probability (IP) theory offers such models, capturing ambiguity and partial belief. This has driven growing interest in imprecise probabilistic machine learning (IPML), where inference and decision-making rely on broader uncertainty models -- highlighting the need for metrics beyond classical probability. This work introduces the Integral Imprecise Probability Metric (IIPM) framework, a Choquet integral-based generalisation of classical Integral Probability Metric (IPM) to the setting of capacities -- a broad class of IP models encompassing many existing ones, including lower probabilities, probability intervals, belief functions, and more. Theoretically, we establish conditions under which IIPM serves as a valid metric and metrises a form of weak convergence of capacities. Practically, IIPM not only enables comparison across different IP models but also supports the quantification of epistemic uncertainty within a single IP model. In particular, by comparing an IP model with its conjugate, IIPM gives rise to a new class of EU measures -- Maximum Mean Imprecision -- which satisfy key axiomatic properties proposed in the Uncertainty Quantification literature. We validate MMI through selective classification experiments, demonstrating strong empirical performance against established EU measures, and outperforming them when classical methods struggle to scale to a large number of classes. Our work advances both theory and practice in IPML, offering a principled framework for comparing and quantifying epistemic uncertainty under imprecision.

Integral Imprecise Probability Metrics

TL;DR

This work introduces Integral Imprecise Probability Metrics (IIPMs), a Choquet-integral generalisation of Integral Probability Metrics (IPMs) to the broad class of imprecise probabilities represented by capacities. It develops Maximum Mean Imputation (MMI), a new epistemic uncertainty (EU) measure derived from distances between a capacity and its conjugate, and demonstrates that IIPMs metrically capture Choquet weak convergence under suitable function classes. Theoretical results include metric properties of IIPM, and practical instantiations such as the Lower Dudley metric and Lower Total Variation, along with connections to the epsilon-contamination Kantorovich problem. Empirically, MMI and its linear-time upper bound (MMI-Lin) outperform traditional EU measures in selective classification across multiclass settings, illustrating both the theory and applicability of IPML for robust uncertainty quantification under imprecision.

Abstract

Quantifying differences between probability distributions is fundamental to statistics and machine learning, primarily for comparing statistical uncertainty. In contrast, epistemic uncertainty (EU) -- due to incomplete knowledge -- requires richer representations than those offered by classical probability. Imprecise probability (IP) theory offers such models, capturing ambiguity and partial belief. This has driven growing interest in imprecise probabilistic machine learning (IPML), where inference and decision-making rely on broader uncertainty models -- highlighting the need for metrics beyond classical probability. This work introduces the Integral Imprecise Probability Metric (IIPM) framework, a Choquet integral-based generalisation of classical Integral Probability Metric (IPM) to the setting of capacities -- a broad class of IP models encompassing many existing ones, including lower probabilities, probability intervals, belief functions, and more. Theoretically, we establish conditions under which IIPM serves as a valid metric and metrises a form of weak convergence of capacities. Practically, IIPM not only enables comparison across different IP models but also supports the quantification of epistemic uncertainty within a single IP model. In particular, by comparing an IP model with its conjugate, IIPM gives rise to a new class of EU measures -- Maximum Mean Imprecision -- which satisfy key axiomatic properties proposed in the Uncertainty Quantification literature. We validate MMI through selective classification experiments, demonstrating strong empirical performance against established EU measures, and outperforming them when classical methods struggle to scale to a large number of classes. Our work advances both theory and practice in IPML, offering a principled framework for comparing and quantifying epistemic uncertainty under imprecision.

Paper Structure

This paper contains 59 sections, 26 theorems, 82 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Integral Probability Metric.
  4. Beyond Precise Probability and Expectations: Capacities and Choquet Integration
  5. Capacities.
  6. Lower Probabilities.
  7. Choquet Integration.
  8. The IIPM framework
  9. Example use cases of the IIPM framework
  10. The Lower Dudley Metric.
  11. Lower Total Variation.
  12. Lower Kantorovich problem with $\epsilon$-contamination sets.
  13. Measuring epistemic uncertainty with IIPM: the MMI
  14. Desirable axiomatic properties for epistemic UQ measure.
  15. A linear-time upper bound.
  16. ...and 44 more sections

Key Result

Lemma 4

For lower probability $\underline{P}$ associated to credal set ${\mathcal{C}}$, we have for any $f\in C_b({\mathcal{X}})$. When $\underline{P}$ is 2-monotonic, the inequality becomes an equality.

Figures (3)

  • Figure 1: Accuracy-Rejection (AR) curves on four classification tasks. The area under the curve (AUC) is reported for numerical comparison. We consistently outperform entropy difference (E-Diff) and match the performance of Generalised Hartley (GH). On large-scale problems, our efficient upper bound (MMI-Lin) remains tractable and continues to outperform E-Diff.
  • Figure 2: A skeleton demonstrating the connection between various uncertainty calculi. "A $\to$ B" means A generalises B, meaning that B is a specific instance of A. The figure is adopted from destercke2008unifying and hullermeier_aleatoric_2021. Most of these frameworks generalise classical probability theory. In the main text, we have discussed capacities, lower and upper probabilities, and standard probabilities in detail, with a brief mention of 2-monotone capacities. Additional discussion of credal sets and belief functions is provided in Appendix A. We do not elaborate on the methods shown in grey; for those, we refer readers to existing surveys and review articles.
  • Figure 3: Comparing the UQ measurements on the withheld test set of the UCI wine dataset aeberhard1992wine. We see that MMI and GH are highly correlated, but not exactly perfectly correlated. The upper bound is also highly correlated, suggesting empirically (along with experiments in the main text), that this approximation is quite tight. Finally, we see that MMI and entropy differences do not correlate that well.

Theorems & Definitions (59)

  • Definition 1: an1933sulla
  • Definition 2: choquet1953
  • Definition 3
  • Definition 4: Choquet Integral
  • Lemma 4
  • Theorem 4
  • Definition 5: Integral Imprecise Probability Metric (IIPM)
  • Corollary 5
  • Proposition 5
  • Theorem 5
  • ...and 49 more