Epistemic Reject Option Prediction

TL;DR

This work addresses high-stakes prediction under finite data by formulating an epistemic reject-option predictor that abstains when epistemic uncertainty, arising from limited data, makes reliable decision-making unlikely. It shifts the learning objective from risk minimization to regret minimization, defining a regret-based reject loss and proving that the epistemic predictor minimizes Bayesian expected regret $R_B^\delta(Q)$ via an epistemic uncertainty threshold $E(x,D)$. The approach unifies and theoretically justifies commonly used uncertainty measures (entropy and variance) as instances of conditional regret under specific losses, and contrasts it with traditional Bayesian rejection that relies on total uncertainty $T(x,D)$. Empirical validation on a synthetic regression task shows the epistemic predictor consistently achieves lower regret (AuReC) across varying data sizes, highlighting its practical potential for selective prediction in data-scarce regimes. The framework thus provides a principled, data-aware mechanism to identify inputs where training data is insufficient for reliable predictions, with implications for safe, deployable AI systems.

Abstract

In high-stakes applications, predictive models must not only produce accurate predictions but also quantify and communicate their uncertainty. Reject-option prediction addresses this by allowing the model to abstain when prediction uncertainty is high. Traditional reject-option approaches focus solely on aleatoric uncertainty, an assumption valid only when large training data makes the epistemic uncertainty negligible. However, in many practical scenarios, limited data makes this assumption unrealistic. This paper introduces the epistemic reject-option predictor, which abstains in regions of high epistemic uncertainty caused by insufficient data. Building on Bayesian learning, we redefine the optimal predictor as the one that minimizes expected regret -- the performance gap between the learned model and the Bayes-optimal predictor with full knowledge of the data distribution. The model abstains when the regret for a given input exceeds a specified rejection cost. To our knowledge, this is the first principled framework that enables learning predictors capable of identifying inputs for which the training data is insufficient to make reliable decisions.

Figures (3)

• Figure 1: The aleatoric reject-option predictor outputs the prediction $h^*(x)$ in regions where the conditional risk $r^*(x)$ does not exceed the rejection cost $\varepsilon$, and abstains otherwise. This example considers the squared loss $\ell(y, \hat{y}) = (y - \hat{y})^2$, the reject cost $\varepsilon=1$ and full knowledge of data distribution $p(x, y)$ defined in Example \ref{['example:1']}. Under this loss, the predictor $h^*(x)$ returns the conditional mean $\mu(x)$, and the conditional risk $r^*(x)$ equals the conditional variance $v(x)$, which is the optimal measure of aleatoric uncertainty.
• Figure 2: Fig \ref{['fig:BayesianRejectOpt']} illustrates the Bayesian reject-option predictor (\ref{['equ:TotalRejectBL']}), which outputs the prediction $H_B(x, D)$ in regions where the total uncertainty $T(x, D)$ does not exceed the rejection cost $\varepsilon = 2$, and abstains otherwise. Fig \ref{['fig:EpistemicRejectOpt']} shows the epistemic reject-option predictor (\ref{['equ:OptEpistRejOptPred']}), which outputs the prediction $H_B(x,D)$ in regions where the epistemic uncertainty $E(x,D)$ does not exceed the rejection cost $\delta=1$, and abstains elsewhere. This example considers the setting of Example \ref{['example:BayesianRejectOption']}, with training data $D$ consisting of $m = 10$ examples drawn from $p(x,y)$ specified in Example \ref{['example:1']}. Under this setup, the predictor is given by $H_B(x, D) = \hbox{\boldmath$x$}^T \hbox{\boldmath$\mu$}_m$, where $\hbox{\boldmath$\mu$}_m$ denotes the mean of the predictive distribution, serving as an estimate of the true mean $\mu(x)$. The total uncertainty is computes as $T(x, D) = \hbox{\boldmath$x$}^T \hbox{\boldmath$\Sigma$}_m \hbox{\boldmath$x$} + v(x)$, which decomposes into the sum of aleatoric and epistemic uncertainties: $T(x, D) = A(x, D) + E(x, D)$, where $A(x, D) = v(x)$ and $E(x, D) = \hbox{\boldmath$x$}^T \hbox{\boldmath$\Sigma$}_m \hbox{\boldmath$x$}$.
• Figure 3: The performance of different reject-option predictors in a synthetic polynomial regression task. The plot shows the Area under the Regret-Coverage (AuReC) curve, where a lower value indicates superior performance, plotted against the number of training samples. The results demonstrate that the proposed predictor using epistemic uncertainty consistently achieves lower regret and outperforms predictors that rely on total or aleatoric uncertainty. Shaded areas indicate the central $20\%$ interval of outcomes across 3000 independent trials.

Theorems & Definitions (7)

• Example 1
• Example 2
• Example 3
• Example 4
• Theorem 1
• Example 5
• Proof 1