Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Strategic Conformal Prediction

Daniel Csillag, Claudio José Struchiner, Guilherme Tegoni Goedert

TL;DR

This work proposes a new framework, Strategic Conformal Prediction, which is capable of robust uncertainty quantification in such a setting, and is backed by a series of theoretical guarantees spanning marginal coverage, training-conditional coverage, tightness and robustness to misspecification that hold in a distribution-free manner.

Abstract

When a machine learning model is deployed, its predictions can alter its environment, as better informed agents strategize to suit their own interests. With such alterations in mind, existing approaches to uncertainty quantification break. In this work we propose a new framework, Strategic Conformal Prediction, which is capable of robust uncertainty quantification in such a setting. Strategic Conformal Prediction is backed by a series of theoretical guarantees spanning marginal coverage, training-conditional coverage, tightness and robustness to misspecification that hold in a distribution-free manner. Experimental analysis further validates our method, showing its remarkable effectiveness in face of arbitrary strategic alterations, whereas other methods break.

Strategic Conformal Prediction

TL;DR

This work proposes a new framework, Strategic Conformal Prediction, which is capable of robust uncertainty quantification in such a setting, and is backed by a series of theoretical guarantees spanning marginal coverage, training-conditional coverage, tightness and robustness to misspecification that hold in a distribution-free manner.

Abstract

When a machine learning model is deployed, its predictions can alter its environment, as better informed agents strategize to suit their own interests. With such alterations in mind, existing approaches to uncertainty quantification break. In this work we propose a new framework, Strategic Conformal Prediction, which is capable of robust uncertainty quantification in such a setting. Strategic Conformal Prediction is backed by a series of theoretical guarantees spanning marginal coverage, training-conditional coverage, tightness and robustness to misspecification that hold in a distribution-free manner. Experimental analysis further validates our method, showing its remarkable effectiveness in face of arbitrary strategic alterations, whereas other methods break.

Paper Structure

This paper contains 31 sections, 20 theorems, 75 equations, 29 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Framework and Method
  3. Construction of the $\Delta$
  4. From an utility-cost decomposition
  5. As a stochastic simulator
  6. Via iterative random search
  7. Extensions of Strategic Conformal Prediction
  8. Additional Guarantees under Modeling Assumptions
  9. Under misspecification of the $\Delta$s
  10. As the $\Delta$ converge
  11. Tightness of the intervals under stability assumptions
  12. Experiments
  13. Conclusion
  14. Proofs
  15. Additional results
  16. ...and 16 more sections

Key Result

Theorem 2.1

Let $Z_1, \ldots, Z_n, Z^\mathrm{test}$ be $n + 1$ exchangeable random variables in $\mathcal{X} \times \mathcal{Y}$. Let $t^\mathrm{strat}_\alpha$ be as above. Then, for any $\alpha \in (0, 1)$,

Figures (29)

  • Figure 1: Coverage and interval sizes for our method versus standard CP. Figures \ref{['fig:coverages']} and \ref{['fig:sizes']} evaluate the coverage and interval sizes under strategic alterations over varying confidence levels $1-\alpha$ on the academic-dropout dataset. Note that our method attains exact coverage, while the baselines have (very) invalid coverage. This comes at a cost of larger predictive sets, but using a better model (Strategic XGBoost) reduces this. For more details and more datasets, see the supplementary material.
  • Figure 2: Coverage of our method and standard CP for varying levels of strategic alterations. The severity ($k_{\max}$) of strategic alterations at test time is indicated in the x-axis, while different curves correspond to conformal calibrations targeting different levels of $k_{\max}$ (along with standard CP) on the academic-dropout dataset, all with $\alpha = 0.1$. The y-axis indicates the test miscoverage under the strategic alterations up to the severity indicated on the x-axis. These curves follow the behaviour guaranteed by Theorems \ref{['thm:conformal-marginal-guarantee']} and \ref{['thm:training-conditional-guarantee']} as well as Proposition \ref{['thm:as-delta-converges']}. Note that using a model more well-suited to strategic alterations "stabilizes" these curves, making them closer together and smoother. For more details and more datasets, see the supplementary material.
  • Figure 3: academic-dropout
  • Figure 4: spambase
  • Figure 5: shoppers
  • ...and 24 more figures

Theorems & Definitions (33)

  • Theorem 2.1
  • Theorem 2.2
  • Theorem 2.3
  • Proposition 2.1
  • Proposition 2.2
  • Proposition 2.3
  • Proposition 2.4
  • Corollary 2.3.1
  • Lemma A.1: Quantile lemma
  • proof
  • ...and 23 more