Scalable Utility-Aware Multiclass Calibration

TL;DR

The paper tackles multiclass calibration by replacing sole reliance on aggregate frequency alignment with utility-aware calibration. It introduces Utility Calibration (UC), a framework that regresses the expected downstream utility $u(f(X),\hat{Y})$ on predictions via $v_u(X)=\langle f(X),\vec{u}(X)\rangle$, and measures calibration error as $\mathrm{UC}(f,u)=\sup_{I} \left|\mathbb{E}[(u(f(X),Y)-v_u(X))\mathbf{1}\{v_u(X)\in I\}]\right|$, thereby unifying and generalizing existing metrics like top-class and class-wise calibration while enabling richer downstream utilities. The authors discuss decision-theoretic implications, showing robustness guarantees to monotone post-processing when UC is small, and they provide scalable estimation and auditing procedures, including a patching-based post-hoc calibration that reduces UC and Brier score. To address scalability, they distinguish proactive and interactive measurability, proposing an interactive approach that samples utilities from a class $\mathcal{U}$ and builds an empirical CDF of UC errors, with finite-sample guarantees and applicability to thousands of classes through finite-dimensional parameterizations (e.g., linear and rank-based utilities). Empirical results on ImageNet-1K and other benchmarks demonstrate that UC-based evaluation reveals nuanced, task-aligned calibration behavior across post-hoc methods and architectures, and that the patching approach yields competitive or superior top-class calibration while preserving interpretability and scalability. Overall, UC provides an application-centric, scalable framework for multiclass calibration that emphasizes reliable downstream decision-making rather than solely minimizing abstract calibration errors.

Abstract

Ensuring that classifiers are well-calibrated, i.e., their predictions align with observed frequencies, is a minimal and fundamental requirement for classifiers to be viewed as trustworthy. Existing methods for assessing multiclass calibration often focus on specific aspects associated with prediction (e.g., top-class confidence, class-wise calibration) or utilize computationally challenging variational formulations. In this work, we study scalable \emph{evaluation} of multiclass calibration. To this end, we propose utility calibration, a general framework that measures the calibration error relative to a specific utility function that encapsulates the goals or decision criteria relevant to the end user. We demonstrate how this framework can unify and re-interpret several existing calibration metrics, particularly allowing for more robust versions of the top-class and class-wise calibration metrics, and, going beyond such binarized approaches, toward assessing calibration for richer classes of downstream utilities.

Figures (7)

• Figure 1: eCDF evaluation for ViT on ImageNet-1K on ${\cal U}_\mathrm{rank}$ and ${\cal U}_\mathrm{lin}$.
• Figure 2: Utility calibration error across iterations for CIFAR100 (left) and ImageNet-1K (right).
• Figure 3: Aligned vs. misaligned utilities across datasets. Top row: utility calibration histories (left: CIFAR100, right: ImageNet-1K). Then: CIFAR100 distribution snapshots at selected iterations (aligned then misaligned), followed by ImageNet-1K distribution snapshots at selected iterations (aligned then misaligned).
• Figure 4: CIFAR10 eCDF plots for ${\cal U}_{\mathrm{lin}}$ and ${\cal U}_{\mathrm{rank}}$.
• Figure 5: CIFAR100 eCDF plots for ${\cal U}_{\mathrm{lin}}$ and ${\cal U}_{\mathrm{rank}}$.

Theorems & Definitions (32)

• Definition 1.1: Mean Calibration Error
• Proposition 3.1: Utility Risk Gap
• Proposition 3.2: Utility Calibration Upper Bounds $\mathrm{DCU}$
• Lemma 3.3: Estimating Utility Calibration Against a Single Function
• Example 3.4: Top-Class and Class-Wise Utilities (${\cal U}_{\mathrm{TCE}}, {\cal U}_{\mathrm{CWE}}$)
• Example 3.5: Linear Utilities (${\cal U}_{\mathrm{lin}}$)
• Example 3.6: Rank-Based and Top-$K$ Utilities (${\cal U}_{\mathrm{rank}}, {\cal U}_{\mathrm{topK}}$)
• Example 3.7: Decision Calibration Utilities (${\cal U}_{\mathrm{dec}, K}$)
• Definition 4.1: Proactive Measurability
• Definition 4.2: Interactive Measurability