Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Calibration for Decision Making

Parikshit Gopalan, Konstantinos Stavropoulos, Kunal Talwar, Pranay Tankala

TL;DR

The paper develops a comprehensive framework for calibrated decision making by introducing CDL$_{\mathcal{K}}$, a quantity measuring maximal post-processing improvement over proper losses, and characterizing when it is tractable by restricting $\mathcal{K}$ to structured families such as generalized monotone post-processings $\mathcal{M}_r$. It shows tight connections between CDL$_{\mathcal{K}}$ and weight-restricted calibration via $\mathsf{thr}'(\mathcal{K})$, yields VC-dimension based sample-complexity results for testing/auditing, and provides reductions to agnostic learning to enable efficient procedures. The work also establishes omniprediction guarantees using Pool Adjacent Violators (PAV) for $\mathcal{M}_+$ and uniform-mass bucketing with recalibration for $\mathcal{M}_r$, thereby giving rigorous performance guarantees for widely used recalibration techniques (isotonic regression, PAV, and bucket-based methods) in ML systems. Overall, the results offer a principled theory guiding the choice of post-processing families to achieve reliable downstream decision-making under calibration constraints, with practical implications for auditing and recalibration pipelines. The framework lays groundwork for efficient CDL analysis and omniprediction in supervised learning, while highlighting fundamental limits for unrestricted Lipschitz post-processing in the offline setting.

Abstract

A decision-theoretic characterization of perfect calibration is that an agent seeking to minimize a proper loss in expectation cannot improve their outcome by post-processing a perfectly calibrated predictor. Hu and Wu (FOCS'24) use this to define an approximate calibration measure called calibration decision loss ($\mathsf{CDL}$), which measures the maximal improvement achievable by any post-processing over any proper loss. Unfortunately, $\mathsf{CDL}$ turns out to be intractable to even weakly approximate in the offline setting, given black-box access to the predictions and labels. We suggest circumventing this by restricting attention to structured families of post-processing functions $K$. We define the calibration decision loss relative to $K$, denoted $\mathsf{CDL}_K$ where we consider all proper losses but restrict post-processings to a structured family $K$. We develop a comprehensive theory of when $\mathsf{CDL}_K$ is information-theoretically and computationally tractable, and use it to prove both upper and lower bounds for natural classes $K$. In addition to introducing new definitions and algorithmic techniques to the theory of calibration for decision making, our results give rigorous guarantees for some widely used recalibration procedures in machine learning.

Efficient Calibration for Decision Making

TL;DR

The paper develops a comprehensive framework for calibrated decision making by introducing CDL, a quantity measuring maximal post-processing improvement over proper losses, and characterizing when it is tractable by restricting to structured families such as generalized monotone post-processings . It shows tight connections between CDL and weight-restricted calibration via , yields VC-dimension based sample-complexity results for testing/auditing, and provides reductions to agnostic learning to enable efficient procedures. The work also establishes omniprediction guarantees using Pool Adjacent Violators (PAV) for and uniform-mass bucketing with recalibration for , thereby giving rigorous performance guarantees for widely used recalibration techniques (isotonic regression, PAV, and bucket-based methods) in ML systems. Overall, the results offer a principled theory guiding the choice of post-processing families to achieve reliable downstream decision-making under calibration constraints, with practical implications for auditing and recalibration pipelines. The framework lays groundwork for efficient CDL analysis and omniprediction in supervised learning, while highlighting fundamental limits for unrestricted Lipschitz post-processing in the offline setting.

Abstract

A decision-theoretic characterization of perfect calibration is that an agent seeking to minimize a proper loss in expectation cannot improve their outcome by post-processing a perfectly calibrated predictor. Hu and Wu (FOCS'24) use this to define an approximate calibration measure called calibration decision loss (), which measures the maximal improvement achievable by any post-processing over any proper loss. Unfortunately, turns out to be intractable to even weakly approximate in the offline setting, given black-box access to the predictions and labels. We suggest circumventing this by restricting attention to structured families of post-processing functions . We define the calibration decision loss relative to , denoted where we consider all proper losses but restrict post-processings to a structured family . We develop a comprehensive theory of when is information-theoretically and computationally tractable, and use it to prove both upper and lower bounds for natural classes . In addition to introducing new definitions and algorithmic techniques to the theory of calibration for decision making, our results give rigorous guarantees for some widely used recalibration procedures in machine learning.

Paper Structure

This paper contains 50 sections, 40 theorems, 152 equations, 2 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Approximate Calibration From Indistinguishability.
  3. Approximate Calibration From No Regret.
  4. Our work:
  5. Overview of Our Results
  6. Definitions
  7. Proper loss functions.
  8. Post-processing functions.
  9. Calibration Decision Loss.
  10. Testing and Auditing
  11. Sample complexity of testing and auditing.
  12. Computationally efficiency from learnability.
  13. Calibration Decision Loss and Weight-Restricted Calibration
  14. Proof Sketch.
  15. Post-Processing and Omniprediction
  16. ...and 35 more sections

Key Result

Theorem 2.6

[theorem]theorem:sample-complexity=intro Let $\mathcal{K}$ be a valid post-processing class, and let $d = \mathsf{VCdim}(\mathsf{thr}(\mathcal{K}))$. Then,

Figures (2)

  • Figure 1: Function $\kappa:[0,1]\to [0,1]$ that crosses every threshold $v\in [0,1]$ of its range at most $3$ times. The monotonicity of the function $\kappa(p)$ changes $14$ times as $p$ grows from $0$ to $1$. Although $\kappa$ is non-monotone, we have $\kappa\in \mathcal{M}_r$ for $r=3$.
  • Figure 2: Visualization of inequalities \ref{['eq:weighted-1-1']} and \ref{['eq:weighted-1-2']}. The dashed line outlines the box $[a, b] \times [a, b]$ in the $(p, y)$-plane. The left and right images correspond to \ref{['eq:weighted-1-1']} and \ref{['eq:weighted-1-2']}, respectively. The inequalities assert that in each image, the blue area is at least as large as the red, minus a $\mathsf{CDL}_\mathcal{K}(J)$ slack term, assuming the marginal distribution of $p$ is uniform.

Theorems & Definitions (90)

  • Definition 2.1: Proper Losses
  • Definition 2.2: Generalized Monotonicity
  • Definition 2.3: Calibration Decision Loss relative to $\mathcal{K}$
  • Definition 2.4: Testing $\mathsf{CDL}$
  • Definition 2.5: Auditing for $\mathsf{CDL}$
  • Theorem 2.6: Sample Complexity Bounds
  • Definition 2.7: Agnostic Learning
  • Theorem 2.8: Testing and Auditing from Agnostic Learning, \ref{['theorem:testing-through-proper-agnostic-learning']}
  • Definition 2.9: Weight-Restricted Calibration Error
  • Theorem 2.10
  • ...and 80 more