Frequentist Validity of Epistemic Uncertainty Estimators

TL;DR

This work proposes a bootstrap-based, frequentist estimator for epistemic uncertainty in predictive models by targeting mutual information between predictions and model parameters. A Bernstein–von Mises–driven expansion links the Bayesian MI to Fisher information, yielding a first-order term $\mathrm{I}(Y_{test};\theta|X_{test},\mathcal{D}_n) \approx \frac{1}{2n}\sum_{k=1}^K \frac{\sigma_k^2}{\hat{p}_k(X_{test};\theta_0)}$, and motivates a simple bootstrap procedure $\mathrm{I}_b$ that consistently estimates MI. The authors show asymptotic equivalence $\mathrm{I} / \mathrm{I}_b \to 1$ and decompose uncertainty into data-sampling and training randomness, offering insight into why deep ensembles perform well. Empirically, the approach validates the plug-in principle, performs comparably to dropout-based MI in active learning, and correlates with deep-ensemble estimates, while enabling data-attribution analyses. The framework provides a theoretically justified, architecture-agnostic route to quantify and interpret epistemic uncertainty in modern deep learning systems.

Abstract

Decomposing prediction uncertainty into its aleatoric (irreducible) and epistemic (reducible) components is critical for the development and deployment of machine learning systems. A popular, principled measure for epistemic uncertainty is the mutual information between the response variable and model parameters. However, evaluating this measure requires access to the posterior distribution of the model parameters, which is challenging to compute. In view of this, we introduce a frequentist measure of epistemic uncertainty based on the bootstrap. Our main theoretical contribution is a novel asymptotic expansion that reveals that our proposed (frequentist) measure and the (Bayesian) mutual information are asymptotically equivalent. This provides frequentist interpretations to mutual information and new computational strategies for approximating it. Moreover, we link our proposed approach to the widely-used heuristic approach of deep ensembles, giving added perspective on their practical success.

Figures (4)

• Figure 1: Teaser Simulation:(left) Training data, true label probability and posterior sample predictions. (center) Predictions from MLEs over bootstrapped datasets. (right) Epistemic uncertainty calculated through Bayesian inference (MCMC) versus our bootstrap based measure.
• Figure 2: (a) Spread under Redraws of Data: Scatter plot of standard deviation in the probability prediction corresponding to the true class for models trained on redraws of the training dataset versus (left) models trained on bootstrapped versions of one dataset and (right) models within a deep ensemble along with Pearson ($\rho$) and Spearman ($r_s$) coefficients. See Sec. \ref{['sec:redraws']} for details. (b) Active Learning: Test accuracy on MNIST for four data acquisition functions. We plot the mean and shade 2 standard deviations across 10 repetitions.
• Figure 3: (a) Epistemic Uncertainty Decomposition: Scatter plot of epistemic uncertainty due to training stochasticity in the bootstrap procedure ($\mathrm{I}^{\textrm{seeds}}_b$) versus (left) epistemic uncertainty due to data resampling in the bootstrap procedure ($\mathrm{I}^{\textrm{resampling}}_b$) and (right) deep ensemble estimate of epistemic uncertainty ($\mathrm{I}^{\textrm{deep ensemble}}_b$). See Sec. \ref{['sec:eu_decomp']} for details. (b) Influence Function Approximation: Estimates of epistemic uncertainty using Dropout, versus using influence functions to approximate predictions from bootstrapped datasets.
• Figure 4: Datamodels Approximation: Datamodel approximation to our measure versus mutual information estimates obtained using Dropout on CIFAR-10 test points. See text for details.

Theorems & Definitions (2)

• Theorem 1
• Theorem 2