Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CLEAR: Calibrated Learning for Epistemic and Aleatoric Risk

Ilia Azizi, Juraj Bodik, Jakob Heiss, Bin Yu

TL;DR

CLEAR, a calibration method with two distinct parameters, $\gamma_1$ and $\gamma_2$, to combine the two uncertainty components and improve the conditional coverage of predictive intervals for regression tasks, is proposed.

Abstract

Accurate uncertainty quantification is critical for reliable predictive modeling. Existing methods typically address either aleatoric uncertainty due to measurement noise or epistemic uncertainty resulting from limited data, but not both in a balanced manner. We propose CLEAR, a calibration method with two distinct parameters, $γ_1$ and $γ_2$, to combine the two uncertainty components and improve the conditional coverage of predictive intervals for regression tasks. CLEAR is compatible with any pair of aleatoric and epistemic estimators; we show how it can be used with (i) quantile regression for aleatoric uncertainty and (ii) ensembles drawn from the Predictability-Computability-Stability (PCS) framework for epistemic uncertainty. Across 17 diverse real-world datasets, CLEAR achieves an average improvement of 28.3\% and 17.5\% in the interval width compared to the two individually calibrated baselines while maintaining nominal coverage. Similar improvements are observed when applying CLEAR to Deep Ensembles (epistemic) and Simultaneous Quantile Regression (aleatoric). The benefits are especially evident in scenarios dominated by high aleatoric or epistemic uncertainty. Project page: https://unco3892.github.io/clear/

CLEAR: Calibrated Learning for Epistemic and Aleatoric Risk

TL;DR

CLEAR, a calibration method with two distinct parameters, and , to combine the two uncertainty components and improve the conditional coverage of predictive intervals for regression tasks, is proposed.

Abstract

Accurate uncertainty quantification is critical for reliable predictive modeling. Existing methods typically address either aleatoric uncertainty due to measurement noise or epistemic uncertainty resulting from limited data, but not both in a balanced manner. We propose CLEAR, a calibration method with two distinct parameters, and , to combine the two uncertainty components and improve the conditional coverage of predictive intervals for regression tasks. CLEAR is compatible with any pair of aleatoric and epistemic estimators; we show how it can be used with (i) quantile regression for aleatoric uncertainty and (ii) ensembles drawn from the Predictability-Computability-Stability (PCS) framework for epistemic uncertainty. Across 17 diverse real-world datasets, CLEAR achieves an average improvement of 28.3\% and 17.5\% in the interval width compared to the two individually calibrated baselines while maintaining nominal coverage. Similar improvements are observed when applying CLEAR to Deep Ensembles (epistemic) and Simultaneous Quantile Regression (aleatoric). The benefits are especially evident in scenarios dominated by high aleatoric or epistemic uncertainty. Project page: https://unco3892.github.io/clear/

Paper Structure

This paper contains 99 sections, 3 theorems, 25 equations, 10 figures, 49 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contributions
  3. Method
  4. Problem scenario
  5. Epistemic uncertainty
  6. Aleatoric uncertainty
  7. CLEAR: combining aleatoric & epistemic uncertainty
  8. Experimental Setup
  9. Data
  10. Baselines
  11. Metrics
  12. Results
  13. Simulations
  14. Real-world data
  15. Empirical comparison with UACQR rossellini2024.
  16. ...and 84 more sections

Key Result

Lemma 2.1

Let $\Lambda$ be compact. Suppose that at least $k$ of the base models used in the PCS ensemble are consistent for the true function $f(x)$, and the quantile regression estimators $\hat{q}_\tau^{\text{ale}}$ are consistent for both $\tau \in \{\alpha/2, 1 - \alpha/2\}$. Then we obtain asymptotic co

Figures (10)

  • Figure 1: Left: Blue represents aleatoric uncertainty, which reflects randomness inherent in the data such as measurement noise. Red represents epistemic uncertainty, which arises from limited sample size. Right: Estimated prediction sets using the CLEAR method, which combines both sources of uncertainty in a data-driven manner.
  • Figure 2: Results for univariate homoskedastic case averaged over 100 simulations: On the left, conditional coverage, and on the right, mean width, for $X=x$. It compares CLEAR, PCS, and ALEATORIC-R (bootstrapped CQR trained on residuals $Y_i - \hat{f}(X_i)$). The dashed horizontal line is the target coverage level of 0.9. CLEAR adapts to maintain target coverage across the input space.
  • Figure 3: Results for real-world data: Quantile loss and NCIW performance of different methods over 10 seeds normalized relative to CLEAR (baseline = 1.0) with error bars are $\pm 1$$\sigma$. Lower values are better. The inset boxplot shows the average (%) relative increase of the metric over CLEAR. EPISTEMIC is PCS-UQ, ALEATORIC is bootstrapped CQR, and ALEATORIC-R uses residuals.
  • Figure 4: Univariate conditional coverage and average width of the prediction intervals for a heteroskedastic case where $\sigma_2(x)=1+|x|$.
  • Figure 5: Univariate conditional coverage and average width of the prediction intervals for a heteroskedastic case where $\sigma_3(x)=1+\frac{1}{1+x^2}$.
  • ...and 5 more figures

Theorems & Definitions (15)

  • Lemma 2.1
  • Definition B.1: Conformalized CLEAR
  • Lemma B.2
  • proof
  • proof : Proof of \ref{['le:AsymptoticConditionalCoverageCLEAR']}
  • Example B.3
  • Example B.4
  • Example B.5
  • Example B.6
  • Example B.7
  • ...and 5 more