Experts Don't Cheat: Learning What You Don't Know By Predicting Pairs

TL;DR

The paper introduces a paired-response training paradigm to separate aleatoric and epistemic uncertainty and to quantify the gap between a model's predicted conditional distribution and the true conditional. By training on pairs $(Y_1,Y_2)$ drawn from $p(Y|X)$ and allowing the model to cheat by conditioning on one sample when predicting the other, it establishes a formal equivalence between second-order calibration and pair-wise calibration, enabling provably-correct frequentist confidence intervals for $p(Y|X)$. The authors prove the cheat-equivalence bijection and derive distribution-free bounds that hold under misspecification, then validate the approach on ambiguous image classification, synthetic language modeling, and safe offline RL with partial observability. Empirically, cheat-corrected pair predictors achieve superior second-order calibration compared to baselines while maintaining first-order calibration and improving safety in downstream tasks. This framework provides a practical path to reliable uncertainty quantification and hallucination detection for complex generative models.

Abstract

Identifying how much a model ${\widehat{p}}_θ(Y|X)$ knows about the stochastic real-world process $p(Y|X)$ it was trained on is important to ensure it avoids producing incorrect or "hallucinated" answers or taking unsafe actions. But this is difficult for generative models because probabilistic predictions do not distinguish between per-response noise (aleatoric uncertainty) and lack of knowledge about the process (epistemic uncertainty), and existing epistemic uncertainty quantification techniques tend to be overconfident when the model underfits. We propose a general strategy for teaching a model to both approximate $p(Y|X)$ and also estimate the remaining gaps between ${\widehat{p}}_θ(Y|X)$ and $p(Y|X)$: train it to predict pairs of independent responses drawn from the true conditional distribution, allow it to "cheat" by observing one response while predicting the other, then measure how much it cheats. Remarkably, we prove that being good at cheating (i.e. cheating whenever it improves your prediction) is equivalent to being second-order calibrated, a principled extension of ordinary calibration that allows us to construct provably-correct frequentist confidence intervals for $p(Y|X)$ and detect incorrect responses with high probability. We demonstrate empirically that our approach accurately estimates how much models don't know across ambiguous image classification, (synthetic) language modeling, and partially-observable navigation tasks, outperforming existing techniques.

Figures (29)

• Figure 1: We train a model (green $\hat{p}_{\theta}$) to predict pairs of i.i.d. ground-truth answers (blue and red ), and allow it to "cheat" by observing one () while predicting the other (). Calibrated models only need to cheat when there is something they don't know, so the amount that the model cheats when its own guesses are presented as expert answers can be used to construct provably-correct "cheat-corrected" estimates of how close $\hat{p}^{\theta}_{{Y\!{|}\!X}}$ is to $p_{{Y\!{|}\!X}}$.
• Figure 2: Each input point $x$ (e.g. an ambiguous image) has its own ground-truth response distribution $p_{{Y\!{|}\!X}}(\cdot|x)$ (e.g. possible human annotator labels for $x$), but first-order calibration only requires the model's prediction $\hat{p}^{\theta}_{{Y\!{|}\!X}}$ to be an average of $p_{{Y\!{|}\!X}}$ across an arbitrary grouping of examples (red and blue), which means $\hat{p}^{\theta}_{{Y\!{|}\!X}}$ can still be far from $p_{{Y\!{|}\!X}}$ for each individual $x$. A second-order-calibrated model additionally measures the suboptimality of this approximation by predicting the per-group covariance$\hat{\bm{\Sigma}}^{\theta}$ of $p_{{Y\!{|}\!X}}$, but this is challenging because $p_{{Y\!{|}\!X}}$ itself is never observed.
• Figure 3: Popular epistemic uncertainty quantification methods are under- or overconfident when $p_{{Y\!{|}\!X}}$ does not match their assumptions. Given a large number of samples $X \in \mathbb{R}, Y \in \{0,1\}$, ensembles and misspecified Gaussian process classifiers report low uncertainty at convergence despite failing to match $p_{{Y\!{|}\!X}}$ around $x \approx 0$; Evidential DL Sensoy2018EvidentialDL reports high uncertainty near $x \approx 2.0$ despite fitting well. In contrast, by using two samples $(Y_1, Y_2)$ for each $X$, our method reports uncertainty that matches the true gap $(\hat{p}^{\theta}_{{Y\!{|}\!X}} - p_{{Y\!{|}\!X}})^2$ even when it underfits.
• Figure 4: Applying \ref{['alg:distfree_bound']} to our model from \ref{['fig:intro_1d_problem']} produces frequentist confidence intervals for $p_{{Y\!{|}\!X}}(y|X)$ which are provably correct with high probability over random $X$. Here $N=10^6, \varepsilon=0.02^2,$ and $\alpha=0.05$; see \ref{['appendix:distnfree_experiment']}.
• Figure 5: Left: For our digits-of-$\pi$ model, binning samples by $C^{\theta}_{{\!\textsc{Cheat}}}$ shows that hallucination rate is usually $\le 1 - C^{\theta}_{{\!\textsc{Cheat}}}$ as predicted by \ref{['thm:conf_hallucination']}, although occasionally $C^{\theta}_{{\!\textsc{Cheat}}} > 1$ due to miscalibration. Right: Ranking samples by $|1 - C^{\theta}_{{\!\textsc{Cheat}}}(y|x)|$ yields a similar or lower hallucination rate than other common filtering strategies when applied to this model.

Theorems & Definitions (34)

• Definition 2.0
• Proposition 2.0
• Proposition 2.0
• Definition 2.0
• Definition 3.0
• Theorem 3.1
• Definition 4.0
• Theorem 4.1
• Definition 4.1
• Proposition 4.1