Do you understand epistemic uncertainty? Think again! Rigorous frequentist epistemic uncertainty estimation in regression

TL;DR

This paper tackles the challenge of distinguishing epistemic from aleatoric uncertainty in regression by introducing a frequentist framework that trains a model to output two conditional predictions per input, where the second prediction conditions on the first. By adopting calibration principles and triplet data $(x,y_1,y_2)$, the authors show that the model covariance ${\rm cov}_\theta(x) = \mathbb{V}[\mathbb{E}[Y|X]\;|\;[x]]$ captures the epistemic component, while the total predictive variance decomposes as $\mathbb{V}_\theta[Y|x] = \mathbb{E}[\mathbb{V}[Y|X]\;|\,[x]] + \mathbb{V}[\mathbb{E}[Y|X]\;|\,[x]]$. They provide a practical training objective using negative log-likelihood for the joint predictor $p_\theta(y_1,y_2|x)$ and show how to estimate covariance via Monte Carlo sampling, all with minimal changes to standard architectures. Validation on synthetic data and real experiments (airfoil aerodynamics and drone noise) demonstrates that ${\rm cov}_\theta(x)$ tracks epistemic uncertainty and grows outside the training domain, supporting a robust, repeatable assessment of prediction reliability in regression settings. The work offers a distinct, non-Bayesian avenue for uncertainty quantification in regression tasks, particularly valuable when repeatable measurements exist and calibration can be achieved.

Abstract

Quantifying model uncertainty is critical for understanding prediction reliability, yet distinguishing between aleatoric and epistemic uncertainty remains challenging. We extend recent work from classification to regression to provide a novel frequentist approach to epistemic and aleatoric uncertainty estimation. We train models to generate conditional predictions by feeding their initial output back as an additional input. This method allows for a rigorous measurement of model uncertainty by observing how prediction responses change when conditioned on the model's previous answer. We provide a complete theoretical framework to analyze epistemic uncertainty in regression in a frequentist way, and explain how it can be exploited in practice to gauge a model's uncertainty, with minimal changes to the original architecture.

Figures (9)

• Figure 1: Estimating epistemic uncertainty. (a) The model is run twice, the first time normally and the second time with the first prediction as a further input. The covariance of the two outputs can be used to quantify the epistemic uncertainty. Heuristically, it can be said that a model that double guesses its own answers presents some degree of epistemic uncertainty.
• Figure 2: A calibrated model can make mistakes. In this example, the model is symmetric around the ordinate and cannot distinguish positive from negative inputs. Even if the target distribution is asymmetric, the model can be calibrated because errors on the opposite sides of the $y$-axis cancel out. Since these errors are due to the model and can in principle be reduced by gathering more data, they are of an epistemic nature.
• Figure 3: Toy model: comparing training on couples and triplets. Results of the simplified experiment. On the left, the model is trained on triplets $(x, y_{1}, y_{2})$, whereas on the right we only used tuples.
• Figure 4: Toy model: results for increasing data corruption. Comparison of epistemic and aleatoric uncertainty for different levels of data corruption. The estimation of the epistemic uncertainty $\mathrm{cov}_\theta$ is unaffected by the level of the aleatoric component $\sigma$.
• Figure 5: Airfoil aerodynamics: result of in-dataset sample. The figure shows the prediction of the model on one of the in-dataset velocities. Both the mean and the variance are well captured. Shaded areas represent the $\sigma$, $2\sigma$ and $3\sigma$ confidence intervals given by the total uncertainty $\sigma_\theta$. The epistemic covariance remains small in the range of $\alpha$ seen during training, and grows rapidly outside of it.

Theorems & Definitions (28)

• Definition 2.1
• Theorem 2.2
• Definition 2.3
• Theorem 2.4
• Theorem 2.5
• Theorem 2.6
• Definition 3.1
• Theorem 3.2
• Proposition 3.3
• Corollary 3.4