An Axiomatic Assessment of Entropy- and Variance-based Uncertainty Quantification in Regression

TL;DR

This work develops an axiomatic framework for uncertainty quantification in regression by embedding predictive distributions in the exponential-family and treating epistemic uncertainty as a second-order distribution over the first-order parameters. It scrutinizes entropy- and variance-based uncertainty measures under this framework, deriving analytic expressions and identifying fundamental strengths and failures (e.g., potentially negative entropy-based uncertainty and sensitivity to parameterization). The study provides theoretical foundations and practical guidance for reliable AU, EU, and TU assessment in regression, highlighting where existing measures align with or violate desirable properties and suggesting directions for future empirical validation and model-generalization. Overall, the paper clarifies how to reason about and compare UQ measures in regression, with implications for designing robust uncertainty representations in safety-critical applications.

Abstract

Uncertainty quantification (UQ) is crucial in machine learning, yet most (axiomatic) studies of uncertainty measures focus on classification, leaving a gap in regression settings with limited formal justification and evaluations. In this work, we introduce a set of axioms to rigorously assess measures of aleatoric, epistemic, and total uncertainty in supervised regression. By utilizing a predictive exponential family, we can generalize commonly used approaches for uncertainty representation and corresponding uncertainty measures. More specifically, we analyze the widely used entropy- and variance-based measures regarding limitations and challenges. Our findings provide a principled foundation for uncertainty quantification in regression, offering theoretical insights and practical guidelines for reliable uncertainty assessment.

Figures (5)

• Figure 1: Comparison of variance and entropy-based uncertainty measures for deep ensembles on a synthetic example. Note in particular the occurrence of negative uncertainty values for the entropy-based measure.
• Figure 2: The figure shows different second-order distributions and their corresponding measures of uncertainty over the parameter $\sigma^2$ of a predictive Gaussian (left, middle) and over the parameter $\lambda$ of a predictive exponential distribution.
• Figure 3: Comparison of variance and entropy-based uncertainty measures for deep evidential regression on a synthetic example. Note in particular the occurrence of negative uncertainty values for the entropy-based measure.
• Figure 4: The figure shows the variance- and entropy-based measures of uncertainty for the deep evidential regression setting, across varying parameters.
• Figure 5: The figure shows the variance- and entropy-based different measures of uncertainty for a deep ensemble $(\mu_m,\sigma_m^2)_{m=1}^2$ with two members across varying parameters.

Theorems & Definitions (25)

• Example 2.1: label=exp:ens
• Example 2.2: label = exp:der
• Example 2.3: continues=exp:ens
• Example 2.4: continues=exp:der
• Definition 3.1
• Proposition 4.1
• Proposition 4.2
• Proposition 4.3
• Proposition 4.4
• Proposition 4.5