Is This Predictor More Informative than Another? A Decision-Theoretical Comparison

TL;DR

This work defines the informativeness gap, $\textsc{InfoGap}(\mu,\nu)$, a decision-theoretic metric comparing arbitrary predictors under heterogeneous payoff tasks. It unifies calibration notions like U-Calibration and CDL and recovers Blackwell informativeness for perfectly calibrated predictors, while extending to miscalibrated ones via a generalized REMD, $\textsc{REMD}^{\text{MisC}}$. A key theoretical contribution is the dual characterization of InfoGap: for perfectly calibrated predictors, $\textsc{InfoGap}(\mu,\nu)=\textsc{REMD}[f_\mu,f_\nu]$, connected to SCDF representations and lattice joins in the space of information structures; miscalibrated predictors are handled through CA-SCDF and a relaxed flow constraint, with strong duality ensuring completeness and soundness. The paper also proves sample-efficient estimation under prediction-only access and demonstrates, through experiments with LLM forecasters on weather and Bitcoin tasks, that InfoGap provides a more decision-relevant assessment than traditional metrics and helps evaluate how calibration post-processing affects downstream payoff.

Abstract

In many real-world applications, a model provider provides probabilistic forecasts to downstream decision-makers who use them to make decisions under diverse payoff objectives. The provider may have access to multiple predictive models, each potentially miscalibrated, and must choose which model to deploy in order to maximize the usefulness of predictions for downstream decisions. A central challenge arises: how can the provider meaningfully compare two predictors when neither is guaranteed to be well-calibrated, and when the relevant decision tasks may differ across users and contexts? To answer this, our first contribution introduces the notion of the informativeness gap between any two predictors, defined as the maximum normalized payoff advantage one predictor offers over the other across all decision-making tasks. Our framework strictly generalizes several existing notions: it subsumes U-Calibration and Calibration Decision Loss, which compare a miscalibrated predictor to its calibrated counterpart, and it recovers Blackwell informativeness as a special case when both predictors are perfectly calibrated. Our second contribution is a dual characterization of the informativeness gap, which gives rise to a natural informativeness measure that can be viewed as a relaxed variant of the earth mover's distance between two prediction distributions. We show that this measure satisfies natural desiderata: it is complete and sound, and it can be estimated sample-efficiently in the prediction-only access setting. We complement our theory with experiments on LLM-based forecasters in real-world prediction tasks, showing that the informativeness gap offers a more decision-relevant alternative to traditional metrics, and provides a principled lens for evaluating how ad hoc calibration post-processing affects downstream decision usefulness.

Figures (9)

• Figure 1: The gray solid line is $S_\mu$ and blue solid line is $S_\nu$ where the predictors $\mu$ and $\nu$ are not dominated (in the Blackwell's sense) with each other. The black dashed line is $S_h$ where $h = \mu\vee \nu$ is the join of the predictors $\mu$ and $\nu$, and $t^* = \mathop{\mathrm{argmax}}\nolimits\nolimits_{t\in[0, 1]}~ S_\mu(t) - S_\nu(t)$. The informativeness gap $0.5\textsc{InfoGap}_{}\!\left[{\mu, \nu}\right]$ is exactly the height of vertical black solid line.
• Figure 2: A graphic illustration of join $h \triangleq \mu \vee \nu$ over two calibrated predictors $\mu, \nu$. The solid arrow implies a Blackwell's order between two predictors, while a dashed arrow not necessarily implies a Blackwell's order. According to \ref{['cor:join equivalency predictor']}, we have that $\textsc{InfoGap}_{}\!\left[{\mu, \nu}\right] = \textsc{InfoGap}_{}\!\left[{h, \nu}\right]$ and $\textsc{InfoGap}_{}\!\left[{\nu, \mu}\right] = \textsc{InfoGap}_{}\!\left[{h, \mu}\right]$.
• Figure 3: IG-tol threshold and pairwise comparability. The plot reports the fraction of LLM pairs that are comparable (red) and strictly comparable (blue) as the IG-tol threshold $s$ increases, aggregated over 10 events (4 LLMs per event, $\binom{4}{2}=6$ pairs; 60 pairs total).
• Figure 4: Brier score and ECE across different LLMs. We report Brier score and expected calibration error (ECE) of four LLM predictors on two representative events: single-day rain event on weather rain dataset (left) and absolute increase threshold of $100 on Bitcoin price dataset (right). Error bars show 95% confidence intervals computed via bootstrap resampling of the obtained $\{p_i, y_i\}$ pairs.
• Figure 5: InfoGap across LLMs. Heatmaps compare the (directional) InfoGap values among four LLMs for the same two events as in Figure \ref{['fig:bs and ece of both']}. Each entry $(i, j)$ reports $\textsc{InfoGap}_{}\!\left[{\mu_i,\mu_j}\right]$, with larger values indicating a larger decision-relevant advantage of $\mu_i$ over $\mu_j$.

Theorems & Definitions (58)

• Example 1.1
• Lemma 2.0
• Definition 2.1: Blackwell informativeness, adopted from B-51B-53
• Lemma 2.1: B-51B-53
• Lemma 2.2: KR-00MS-06
• Definition 2.2: Informativeness gap
• Proposition 2.2
• Lemma 2.2
• Definition 3.1: EMD between distributions
• Definition 3.2: Relaxed EMD between distributions