Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Data-Adaptive Tradeoffs among Multiple Risks in Distribution-Free Prediction

Drew T. Nguyen, Reese Pathak, Anastasios N. Angelopoulos, Stephen Bates, Michael I. Jordan

TL;DR

This work develops methods that permit valid control of risk when threshold and tradeoff parameters are chosen adaptively and supports monotone and nearly-monotone risks, but otherwise makes no distributional assumptions.

Abstract

Decision-making pipelines are generally characterized by tradeoffs among various risk functions. It is often desirable to manage such tradeoffs in a data-adaptive manner. As we demonstrate, if this is done naively, state-of-the art uncertainty quantification methods can lead to significant violations of putative risk guarantees. To address this issue, we develop methods that permit valid control of risk when threshold and tradeoff parameters are chosen adaptively. Our methodology supports monotone and nearly-monotone risks, but otherwise makes no distributional assumptions. To illustrate the benefits of our approach, we carry out numerical experiments on synthetic data and the large-scale vision dataset MS-COCO.

Data-Adaptive Tradeoffs among Multiple Risks in Distribution-Free Prediction

TL;DR

This work develops methods that permit valid control of risk when threshold and tradeoff parameters are chosen adaptively and supports monotone and nearly-monotone risks, but otherwise makes no distributional assumptions.

Abstract

Decision-making pipelines are generally characterized by tradeoffs among various risk functions. It is often desirable to manage such tradeoffs in a data-adaptive manner. As we demonstrate, if this is done naively, state-of-the art uncertainty quantification methods can lead to significant violations of putative risk guarantees. To address this issue, we develop methods that permit valid control of risk when threshold and tradeoff parameters are chosen adaptively. Our methodology supports monotone and nearly-monotone risks, but otherwise makes no distributional assumptions. To illustrate the benefits of our approach, we carry out numerical experiments on synthetic data and the large-scale vision dataset MS-COCO.
Paper Structure (36 sections, 17 theorems, 88 equations, 10 figures)

This paper contains 36 sections, 17 theorems, 88 equations, 10 figures.

Table of Contents

  1. Introduction
  2. Case study: risk tradeoffs and risk control on MS-COCO
  3. Prior work
  4. Risk Control with Uniform Bounds
  5. Setting
  6. Finite-sample result for monotone losses
  7. Asymptotic result for monotone losses
  8. Extension to localized, simultaneous risk control
  9. Extension to non-monotone losses
  10. Combinations of monotone risks
  11. General approach:
  12. Illustration for selective classification:
  13. Nearly monotone risks
  14. Monotonizing the loss:
  15. Batch-and-monotonize:
  16. ...and 21 more sections

Key Result

Theorem 2.1

For every $\lambda > 0$, we have

Figures (10)

  • Figure 1: Histograms for 20K realizations of the risk gap $\alpha - \texttt{FNR}$ under a data-dependent choice of $\alpha$, given observations of $n$ calibration points. Each histogram represents different ways to set $\alpha$. The solid outline has total area $\delta = 0.1$.
  • Figure 2: Prediction sets on an example from MS-COCO using the classifier $\mathcal{C}_t$ of Section \ref{['sec:violation']}, as the threshold $t$ is varied. Also plotted is the FNR and FPR averaged over 60K held-out images, which are unobserved in practice; additionally, a rug plot of the scheme $t = \hat{t}$ optimized as in equation \ref{['eq:eventradeoff']}, based on simulation draws of size $n = 500$.
  • Figure 3: Simulated data: Log-log plot of true quantile of $D_n$ and its median bootstrap estimate (and 90/10% quantiles) computed from 3K Monte Carlo runs.
  • Figure 4: Simulated data: Anywhere miscoverage based on 20K Monte Carlo runs.
  • Figure 5: Simulated data: Selected set miscoverage based on 20K Monte Carlo runs.
  • ...and 5 more figures

Theorems & Definitions (26)

  • Theorem 2.1
  • Corollary 2.1: Nonasymptotic confidence bounds
  • Theorem 2.2
  • Lemma 1
  • Corollary 2.2: Confidence bounds via risk resampling
  • Theorem 2.3: Confidence bound via restricted risk resampling
  • Proposition 4.1
  • Proposition 4.2
  • Theorem 4.1
  • Theorem A.1: Based on Theorem 3 of waudby-smithEstimatingMeansBounded2023
  • ...and 16 more