Quantifying Aleatoric and Epistemic Uncertainty with Proper Scoring Rules
Paul Hofman, Yusuf Sale, Eyke Hüllermeier
TL;DR
This work targets robust uncertainty quantification in ML by reframing aleatoric and epistemic uncertainty through strictly proper scoring rules. It unifies two common epistemic representations—second-order distributions (Bayesian view) and credal sets (Levi view)—and derives new, scoring-rule–driven measures for both Bayesian and Levi agents. By replacing MI-based analyses with loss-based decompositions, the approach yields flexible instantiations across different scoring rules (e.g., log-loss, zero-one loss) and accommodates a broader class of uncertainty representations. The results offer a principled, versatile framework for uncertainty quantification applicable to safety-critical ML systems, with planned theoretical and empirical extensions.
Abstract
Uncertainty representation and quantification are paramount in machine learning and constitute an important prerequisite for safety-critical applications. In this paper, we propose novel measures for the quantification of aleatoric and epistemic uncertainty based on proper scoring rules, which are loss functions with the meaningful property that they incentivize the learner to predict ground-truth (conditional) probabilities. We assume two common representations of (epistemic) uncertainty, namely, in terms of a credal set, i.e. a set of probability distributions, or a second-order distribution, i.e., a distribution over probability distributions. Our framework establishes a natural bridge between these representations. We provide a formal justification of our approach and introduce new measures of epistemic and aleatoric uncertainty as concrete instantiations.