Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bayesian Conformal Prediction as a Decision Risk Problem

Fanyi Wu, Veronika Lohmanova, Samuel Kaski, Michele Caprio

TL;DR

The paper addresses uncertainty quantification under model misspecification by reframing conformal prediction as a decision-risk problem and optimising the split CP threshold under conformal risk control. It introduces optimised Bayesian Conformal Prediction (BCP), which uses Bayesian posterior predictive densities as non-conformity scores via AOI sampling and applies Bayesian Quadrature to efficiently estimate the expected prediction-set size under the input distribution, while enforcing finite-sample validity with an $L^+$-based CRC bound. Empirically, BCP achieves near-nominal coverage across regression, binary classification, and distribution-shifted ImageNet-A settings, with prediction sets that are competitive in size and markedly more stable across data splits than purely Bayesian intervals or classical split CP. This framework demonstrates how Bayesian structure can improve efficiency and stability without sacrificing guarantee of coverage, offering robust uncertainty quantification in practical, misspecified environments.

Abstract

Bayesian posterior predictive densities as non-conformity scores and Bayesian quadrature are used to estimate and minimise the expected prediction set size. Operating within a split conformal framework, BCP provides valid coverage guarantees and demonstrates reliable empirical coverage under model misspecification. Across regression and classification tasks, including distribution-shifted settings such as ImageNet-A, BCP yields prediction sets of comparable size to split conformal prediction, while exhibiting substantially lower run-to-run variability in set size. In sparse regression with nominal coverage of 80 percent, BCP achieves 81 percent empirical coverage under a misspecified prior, whereas Bayesian credible intervals under-cover at 49 percent.

Bayesian Conformal Prediction as a Decision Risk Problem

TL;DR

The paper addresses uncertainty quantification under model misspecification by reframing conformal prediction as a decision-risk problem and optimising the split CP threshold under conformal risk control. It introduces optimised Bayesian Conformal Prediction (BCP), which uses Bayesian posterior predictive densities as non-conformity scores via AOI sampling and applies Bayesian Quadrature to efficiently estimate the expected prediction-set size under the input distribution, while enforcing finite-sample validity with an -based CRC bound. Empirically, BCP achieves near-nominal coverage across regression, binary classification, and distribution-shifted ImageNet-A settings, with prediction sets that are competitive in size and markedly more stable across data splits than purely Bayesian intervals or classical split CP. This framework demonstrates how Bayesian structure can improve efficiency and stability without sacrificing guarantee of coverage, offering robust uncertainty quantification in practical, misspecified environments.

Abstract

Bayesian posterior predictive densities as non-conformity scores and Bayesian quadrature are used to estimate and minimise the expected prediction set size. Operating within a split conformal framework, BCP provides valid coverage guarantees and demonstrates reliable empirical coverage under model misspecification. Across regression and classification tasks, including distribution-shifted settings such as ImageNet-A, BCP yields prediction sets of comparable size to split conformal prediction, while exhibiting substantially lower run-to-run variability in set size. In sparse regression with nominal coverage of 80 percent, BCP achieves 81 percent empirical coverage under a misspecified prior, whereas Bayesian credible intervals under-cover at 49 percent.
Paper Structure (36 sections, 1 theorem, 39 equations, 5 figures, 4 tables, 2 algorithms)

This paper contains 36 sections, 1 theorem, 39 equations, 5 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. CP as a Decision Risk Problem
  4. Method
  5. AOI posterior sampling
  6. Bayesian risk optimisation
  7. Bayesian Quadrature
  8. Empirical risk control via $L^+$.
  9. High-Level View of BCP.
  10. Experiments
  11. Sparse Regression on the Diabetes Dataset
  12. BCP refinement.
  13. Breast Cancer Classification
  14. High-dimensional ImageNet-A Classification
  15. Qualitative comparison and scoring methods.
  16. ...and 21 more sections

Key Result

Proposition 1

Assume that the calibration and test samples are exchangeable conditional on the training data, and that the score function eq:bayes-score-split is fixed with respect to the calibration labels. Let $\lambda$ be selected using conformal risk control applied to the resulting calibration losses. Then t

Figures (5)

  • Figure 1: Conceptual overview of BCP. It combines a Bayesian approach with CP through (i) Bayesian non-conformity scores constructed via AOI sampling, and (ii) decision-risk optimisation based on BQ under CRC.
  • Figure 2: BQ optimisation of the threshold $\lambda$ under a coverage constraint. The blue curve shows the expected prediction set size $\mathbb{E}_x \left[|\mathcal{C}(x; \theta, \lambda)|\right]$ as a function of $\lambda$, increasing as more $y$'s are included. The green dashed line marks the boundary induced by the CRC constraint, beyond which the target coverage level is satisfied with confidence $1-\beta$. The optimal $\lambda^\ast$ is selected as the smallest $\lambda$ that ensures sufficient coverage while minimising prediction set size.
  • Figure 3: Comparison of prediction regions from CB and BCP methods on a regression task. (a) Empirical coverage versus average interval width for four conformal methods. The dashed line indicates the target coverage level of $80\%$. (b) Predicted intervals for a random subset of test points from each method, showing interval width and coverage. (c) Histograms of empirical coverage over multiple random splits for CB and BCP, with the dashed line indicating the target coverage level. (d) Violin plots summarising the same distributions, where horizontal red and green markers denote the median and mean, respectively.
  • Figure 4: Empirical coverage of Bayesian conformal prediction (BCP) on the breast cancer classification task. (a) Coverage as a function of the target risk level $\alpha$ for different HPD levels $\beta$. (b) Coverage as a function of $\beta$ for different values of $\alpha$. The dashed horizontal line indicates the target coverage level $1-\alpha$.
  • Figure 5: ImageNet-A demo results ($\alpha=0.2$). (a) Prediction set size distributions on the ImageNet-A test set for Split-CP, CB, and BCP ($\alpha=0.2$). Dashed lines denote the mean set size of each method. (b) Qualitative comparison on an ImageNet-A test example (index 795, ground truth: grasshopper). We show the top-ranked labels and associated scores produced by MSP/CB and BCP. The star indicates the ground-truth class.

Theorems & Definitions (1)

  • Proposition 1: Validity of Bayesian Scores under Split CP