CBMA: Improving conformal prediction through Bayesian model averaging

TL;DR

This paper tackles the problem of obtaining efficiently narrow conformal prediction sets when multiple Bayesian models may be plausible. It introduces Conformal Bayesian model averaging (CBMA), which aggregates per-model conformity scores using data-driven weights derived from posterior model probabilities and predictive densities, forming a single valid conformity measure. The authors prove that CBMA is exchangeable and that, if the true model lies in the model space, CBMA prediction sets converge to the optimal conformal Bayes set as sample size grows, thereby achieving asymptotic efficiency under model uncertainty. Empirical results on simulated quadratic and heteroskedastic settings, plus a California housing data example, show CBMA yields valid coverage with shorter intervals than competing approaches, highlighting its practical value for robust uncertainty quantification in uncertain-model regimes.

Abstract

Conformal prediction has emerged as a popular technique for facilitating valid predictive inference across a spectrum of machine learning models, under minimal assumption of exchangeability. Recently, Hoff (2023) showed that full conformal Bayes provides the most efficient prediction sets (smallest by expected volume) among all prediction sets that are valid at the $(1 - α)$ level if the model is correctly specified. However, a critical issue arises when the Bayesian model itself may be mis-specified, resulting in prediction set that might be suboptimal, even though it still enjoys the frequentist coverage guarantee. To address this limitation, we propose an innovative solution that combines Bayesian model averaging (BMA) with conformal prediction. This hybrid not only leverages the strengths of Bayesian conformal prediction but also introduces a layer of robustness through model averaging. Theoretically, we prove that the resulting prediction set will converge to the optimal level of efficiency, if the true model is included among the candidate models. This assurance of optimality, even under potential model uncertainty, provides a significant improvement over existing methods, ensuring more reliable and precise uncertainty quantification.

Figures (6)

• Figure 1: Quadratic model: Comparison of CBMA and CBM1 for different sample sizes.
• Figure 2: Approximation using Hermite polynomials: Comparison of mean length for prediction sets obtained with different methods: CBMA (proposed method), BMA, individual Bayes prediction sets (in red), and individual conformal Bayes (CB) prediction sets (in blue). Here, we report results based on $E = 50$ number of experiments. We have set the target coverage as $(1 - \alpha) = 0.80$, sample size $n = 100$ and $\theta = 1$.
• Figure 3: California Housing data: comparison of mean lengths of intervals using Bayes prediction, conformal Bayes (CB) for all four models, CBMA and BMA for different sample sizes.
• Figure 4: Posterior model probabilities (PMP) for all three models considered in our experiment with quadratic model with sample size $n= 100$.
• Figure 5: Approximation using Hermite polynomials: Comparison of mean coverage for prediction sets obtained with different methods: CBMA (proposed method), BMA, individual Bayes prediction sets (in red), and individual conformal Bayes (CB) prediction sets (in blue). Here, we report results based on $E = 50$ number of experiments. We have set the target coverage as $(1 - \alpha) = 0.80$, sample size $n = 100$ and $\theta = 1$.

Theorems & Definitions (6)

• Lemma 1
• Theorem 1
• Theorem 2
• Remark 1
• Remark 2
• Remark 3