Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generative Conformal Prediction with Vectorized Non-Conformity Scores

Minxing Zheng, Shixiang Zhu

TL;DR

This work addresses the conservatism of conformal prediction in multi-dimensional settings by introducing Generative Conformal Prediction with Vectorized Non-Conformity Scores (GCP-VCR). It leverages a generative model to sample multiple conditional predictions, forms a vector of non-conformity scores across ranked samples, and optimizes rank-specific quantiles to create density-adaptive uncertainty balls. The approach comes with theoretical validity guarantees and demonstrates superior efficiency over state-of-the-art baselines on synthetic, MNIST-like, and real datasets, especially in multimodal and complex distributions. Overall, GCP-VCR delivers more flexible, data-adaptive uncertainty sets while preserving the guaranteed coverage, enabling more informative decision-making in high-stakes or complex predictive tasks.

Abstract

Conformal prediction (CP) provides model-agnostic uncertainty quantification with guaranteed coverage, but conventional methods often produce overly conservative uncertainty sets, especially in multi-dimensional settings. This limitation arises from simplistic non-conformity scores that rely solely on prediction error, failing to capture the prediction error distribution's complexity. To address this, we propose a generative conformal prediction framework with vectorized non-conformity scores, leveraging a generative model to sample multiple predictions from the fitted data distribution. By computing non-conformity scores across these samples and estimating empirical quantiles at different density levels, we construct adaptive uncertainty sets using density-ranked uncertainty balls. This approach enables more precise uncertainty allocation -- yielding larger prediction sets in high-confidence regions and smaller or excluded sets in low-confidence regions -- enhancing both flexibility and efficiency. We establish theoretical guarantees for statistical validity and demonstrate through extensive numerical experiments that our method outperforms state-of-the-art techniques on synthetic and real-world datasets.

Generative Conformal Prediction with Vectorized Non-Conformity Scores

TL;DR

This work addresses the conservatism of conformal prediction in multi-dimensional settings by introducing Generative Conformal Prediction with Vectorized Non-Conformity Scores (GCP-VCR). It leverages a generative model to sample multiple conditional predictions, forms a vector of non-conformity scores across ranked samples, and optimizes rank-specific quantiles to create density-adaptive uncertainty balls. The approach comes with theoretical validity guarantees and demonstrates superior efficiency over state-of-the-art baselines on synthetic, MNIST-like, and real datasets, especially in multimodal and complex distributions. Overall, GCP-VCR delivers more flexible, data-adaptive uncertainty sets while preserving the guaranteed coverage, enabling more informative decision-making in high-stakes or complex predictive tasks.

Abstract

Conformal prediction (CP) provides model-agnostic uncertainty quantification with guaranteed coverage, but conventional methods often produce overly conservative uncertainty sets, especially in multi-dimensional settings. This limitation arises from simplistic non-conformity scores that rely solely on prediction error, failing to capture the prediction error distribution's complexity. To address this, we propose a generative conformal prediction framework with vectorized non-conformity scores, leveraging a generative model to sample multiple predictions from the fitted data distribution. By computing non-conformity scores across these samples and estimating empirical quantiles at different density levels, we construct adaptive uncertainty sets using density-ranked uncertainty balls. This approach enables more precise uncertainty allocation -- yielding larger prediction sets in high-confidence regions and smaller or excluded sets in low-confidence regions -- enhancing both flexibility and efficiency. We establish theoretical guarantees for statistical validity and demonstrate through extensive numerical experiments that our method outperforms state-of-the-art techniques on synthetic and real-world datasets.

Paper Structure

This paper contains 24 sections, 2 theorems, 22 equations, 9 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Proposed Method
  5. Vectorized Non-Conformity Score
  6. Efficiency-Enhanced Prediction Set
  7. Quantile Optimization
  8. Comparison with PCP
  9. Experiments
  10. Baselines:
  11. Implementation Configuration:
  12. Synthetic Data
  13. MNIST: Digit Image Generation
  14. Real Data
  15. Discussion and Conclusion
  16. ...and 9 more sections

Key Result

Theorem 3.1

For testing data $(X_{n+1},Y_{n+1})$ that is exchangeable with calibration $\mathcal{D}$, the prediction set $\hat{\mathcal{C}}_\text{VCR}(X_{n+1};\beta^*)$ provides valid coverage, i.e.,

Figures (9)

  • Figure 1: An illustrative comparison betwen the proposed GCP-VCR and PCP, where GCP-VCR provides a more flexible and efficient prediction set. The non-conformity score vector $(E_{i,1}, \dots, E_{i,K})^\top$ provides more detailed information on the underlying distribution represented by the blue area. Samples in denser regions (darker blue) are ranked higher (small $r$), indicating greater model confidence and larger $\mathcal{C}_r$ sizes, while low-rank scores (large $r$) capture uncertainty in sparser regions, where $\mathcal{C}_r$ sizes are smaller.
  • Figure 2: Demonstration of the proposed approximated algorithm. Subplot \ref{['fig:approximated_solution_demo_subplot_a']} presents the efficiency of approximated solutions with different initializations toward optimal efficiency over iterations. Subplot \ref{['fig:approximated_solution_demo_subplot_b']} presents the solution paths of the approximated solutions for different initialization, with arrows indicating the path direction over iterations. The shaded area is the feasible region of the optimization problem, and the darker regions correspond to higher efficiency.
  • Figure 3: Comparison of MNIST images sampled from prediction sets of GCP-VCR and PCP, respectively. Each column corresponds to different values of $X$. The randomly sampled images of GCP-VCR align more closely with the target $X$ compared to PCP, indicating tighter prediction sets with fewer incorrect images.
  • Figure 4: Comparison of prediction sets across four methods that have the highest efficiency on Bike dataset. The colored area indicates the prediction set, and the red marker is the target location.
  • Figure 5: Feasible region and optimal solution of optimization problem in \ref{['eq:optim_problem']} for $K=2$. The colored region shows the feasible region of $\beta$ with darker color indicating higher efficiency and the red point shows the position of $\beta^*$.
  • ...and 4 more figures

Theorems & Definitions (7)

  • Theorem 3.1: Validity of GCP-VCR
  • Remark 3.2
  • Corollary 3.3: Validity of Approximated Solution
  • Definition B.1: Vector partial order
  • proof
  • proof
  • Remark B.2