Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online Selective Conformal Prediction with Asymmetric Rules: A Permutation Test Approach

Mingyi Zheng, Ying Jin

TL;DR

This work introduces PEMI, a permutation-based Mondrian conformal inference framework for online selective prediction under arbitrary asymmetric selection rules. By calibrating conformity scores over a reference set of data permutations that preserve the selection event, PEMI achieves finite-sample selection-conditional coverage under exchangeability, for both full and Monte-Carlo permutation schemes; it also extends to incorporate offline data and multiple test samples. The authors provide theoretical guarantees and develop computationally efficient instantiations for covariate-based, conformal p/e-value based, and early-outcome-based selection rules, with applications to drug discovery and extensive simulation studies. The approach broadens the scope of selective conformal prediction beyond symmetric settings, reduces vacuity in prediction sets, and offers practical tools for online uncertainty quantification and FCR control under a taxonomy of selection rules.

Abstract

Selective conformal prediction aims to construct prediction sets with valid coverage for a test unit conditional on it being selected by a data-driven mechanism. While existing methods in the offline setting handle any selection mechanism that is permutation invariant to the labeled data, their extension to the online setting -- where data arrives sequentially and later decisions depend on earlier ones -- is challenged by the fact that the selection mechanism is naturally asymmetric. As such, existing methods only address a limited collection of selection mechanisms. In this paper, we propose PErmutation-based Mondrian Conformal Inference (PEMI), a general permutation-based framework for selective conformal prediction with arbitrary asymmetric selection rules. Motivated by full and Mondrian conformal prediction, PEMI identifies all permutations of the observed data (or a Monte-Carlo subset thereof) that lead to the same selection event, and calibrates a prediction set using conformity scores over this selection-preserving reference set. Under standard exchangeability conditions, our prediction sets achieve finite-sample exact selection-conditional coverage for any asymmetric selection mechanism and any prediction model. PEMI naturally incorporates additional offline labeled data, extends to selection mechanisms with multiple test samples, and achieves FCR control with fine-grained selection taxonomies. We further work out several efficient instantiations for commonly-used online selection rules, including covariate-based rules, conformal p/e-values-based procedures, and selection based on earlier outcomes. Finally, we demonstrate the efficacy of our methods across various selection rules on a real drug discovery dataset and investigate their performance via simulations.

Online Selective Conformal Prediction with Asymmetric Rules: A Permutation Test Approach

TL;DR

This work introduces PEMI, a permutation-based Mondrian conformal inference framework for online selective prediction under arbitrary asymmetric selection rules. By calibrating conformity scores over a reference set of data permutations that preserve the selection event, PEMI achieves finite-sample selection-conditional coverage under exchangeability, for both full and Monte-Carlo permutation schemes; it also extends to incorporate offline data and multiple test samples. The authors provide theoretical guarantees and develop computationally efficient instantiations for covariate-based, conformal p/e-value based, and early-outcome-based selection rules, with applications to drug discovery and extensive simulation studies. The approach broadens the scope of selective conformal prediction beyond symmetric settings, reduces vacuity in prediction sets, and offers practical tools for online uncertainty quantification and FCR control under a taxonomy of selection rules.

Abstract

Selective conformal prediction aims to construct prediction sets with valid coverage for a test unit conditional on it being selected by a data-driven mechanism. While existing methods in the offline setting handle any selection mechanism that is permutation invariant to the labeled data, their extension to the online setting -- where data arrives sequentially and later decisions depend on earlier ones -- is challenged by the fact that the selection mechanism is naturally asymmetric. As such, existing methods only address a limited collection of selection mechanisms. In this paper, we propose PErmutation-based Mondrian Conformal Inference (PEMI), a general permutation-based framework for selective conformal prediction with arbitrary asymmetric selection rules. Motivated by full and Mondrian conformal prediction, PEMI identifies all permutations of the observed data (or a Monte-Carlo subset thereof) that lead to the same selection event, and calibrates a prediction set using conformity scores over this selection-preserving reference set. Under standard exchangeability conditions, our prediction sets achieve finite-sample exact selection-conditional coverage for any asymmetric selection mechanism and any prediction model. PEMI naturally incorporates additional offline labeled data, extends to selection mechanisms with multiple test samples, and achieves FCR control with fine-grained selection taxonomies. We further work out several efficient instantiations for commonly-used online selection rules, including covariate-based rules, conformal p/e-values-based procedures, and selection based on earlier outcomes. Finally, we demonstrate the efficacy of our methods across various selection rules on a real drug discovery dataset and investigate their performance via simulations.
Paper Structure (54 sections, 11 theorems, 109 equations, 13 figures, 3 algorithms)

This paper contains 54 sections, 11 theorems, 109 equations, 13 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Our approach: selective inference over permutations
  3. Preview of results
  4. Related work
  5. General method
  6. Warm-up: full conformal prediction as permutation test
  7. Permutation FCP with asymmetric score.
  8. PEMI: General procedure
  9. Choice of $\Pi_t^\star$.
  10. Theoretical guarantees
  11. Incorporating offline data
  12. General selection taxonomy and FCR control
  13. FCR Control.
  14. Generalization to multiple test samples
  15. Computationally efficient instantiations
  16. ...and 39 more sections

Key Result

Theorem 1

Suppose the data $\{Z_i\}_{i=1}^t$ are exchangeable. For any selection rule ${\mathcal{S}}_t\colon \mathcal{Z}^{t-1}\times \mathcal{X}\to \{0,1\}$ and any conformity score $\mathcal{V}_t\colon \mathcal{Z}^{t}\to \mathbb{R}$, and for $\Pi_t^\star \in\{\Pi_t,\Pi_t^{(M)}\}$ stated above, we have P( Y_t P( Y_t∈ C_α,t^rand (Π_t^⋆) | S_t=1) = 1-α, ∀ t≥ 1. Notably, in the case $\Pi_t^\star = \Pi_t^{

Figures (13)

  • Figure 1: An illustration of the proposed PEMI method. Starting from a set of permutations $\Pi$ of the observed data, we identify the subset which leads to the same selection event after permutation. The final prediction set is then calibrated based on the subset of permutations.
  • Figure 2: Selection-conditional coverage (left), median prediction set size (middle), and the fraction of prediction sets with infinite length (right) in drug property prediction under a decision-driven selection rule for PEMI (Ours Upper), randomized PEMI (Ours randomize), vanilla conformal prediction (Vanilla), EXPRESS (EXPRESS) and OnlineSCI with three parameter choices (OnlineSCI1/2/3). The purple dashed line is the target coverage level $1-\alpha=0.6$.
  • Figure 3: Results under a selection rule based on weighted quantile of predictions when applying PEMI (Ours Upper), randomized PEMI (Ours randomize), vanilla conformal prediction (Vanilla) and OnlineSCI OnlineSCI). The purple dashed line is the target coverage level $1-\alpha=0.6$. Details are otherwise as Figure \ref{['fig:real decision']}.
  • Figure 4: Results under selection based on weighted average of earlier predictions; details otherwise as Figure \ref{['fig:real wq of mu']}.
  • Figure 5: Results under selection based on model uncertainty with online optimization; details otherwise as Figure \ref{['fig:real wq of mu']}.
  • ...and 8 more figures

Theorems & Definitions (12)

  • Remark 1
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Proposition 1
  • Proposition 2
  • Proposition 3
  • Proposition 4
  • Proposition 5
  • ...and 2 more