Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multivariate Conformal Prediction via Conformalized Gaussian Scoring

Sacha Braun, Eugène Berta, Michael I. Jordan, Francis Bach

TL;DR

This work tackles the challenge of achieving conditional coverage in multivariate conformal prediction by introducing Gaussian-conformal prediction, which treats ${\mathbb P}_{Y|X}$ as a feature-dependent Gaussian ${\mathcal N}(f(X),\Sigma(X))$. It derives a closed-form, Mahalanobis-distance-based non-conformity score, enabling ellipsoidal conformal sets that adapt to local uncertainty and heteroskedasticity, and extends naturally to missing outputs, partially revealed outputs, and transformations of the output space. The method can be applied post-hoc on top of existing predictors by learning a conditional covariance, provides finite-sample coverage guarantees (marginal, with extensions to the transformed and partially observed cases), and shows improved empirical conditional coverage in synthetic and real multivariate datasets. Practically, this yields data-dependent, computationally efficient conformal sets suitable for complex, high-dimensional prediction tasks with incomplete or evolving output information. Overall, the approach offers a flexible, scalable framework that enhances uncertainty quantification for multivariate predictions in conformal prediction settings.

Abstract

While achieving exact conditional coverage in conformal prediction is unattainable without making strong, untestable regularity assumptions, the promise of conformal prediction hinges on finding approximations to conditional guarantees that are realizable in practice. A promising direction for obtaining conditional dependence for conformal sets--in particular capturing heteroskedasticity--is through estimating the conditional density $\mathbb{P}_{Y|X}$ and conformalizing its level sets. Previous work in this vein has focused on nonconformity scores based on the empirical cumulative distribution function (CDF). Such scores are, however, computationally costly, typically requiring expensive sampling methods. To avoid the need for sampling, we observe that the CDF-based score reduces to a Mahalanobis distance in the case of Gaussian scores, yielding a closed-form expression that can be directly conformalized. Moreover, the use of a Gaussian-based score opens the door to a number of extensions of the basic conformal method; in particular, we show how to construct conformal sets with missing output values, refine conformal sets as partial information about $Y$ becomes available, and construct conformal sets on transformations of the output space. Finally, empirical results indicate that our approach produces conformal sets that more closely approximate conditional coverage in multivariate settings compared to alternative methods.

Multivariate Conformal Prediction via Conformalized Gaussian Scoring

TL;DR

This work tackles the challenge of achieving conditional coverage in multivariate conformal prediction by introducing Gaussian-conformal prediction, which treats as a feature-dependent Gaussian . It derives a closed-form, Mahalanobis-distance-based non-conformity score, enabling ellipsoidal conformal sets that adapt to local uncertainty and heteroskedasticity, and extends naturally to missing outputs, partially revealed outputs, and transformations of the output space. The method can be applied post-hoc on top of existing predictors by learning a conditional covariance, provides finite-sample coverage guarantees (marginal, with extensions to the transformed and partially observed cases), and shows improved empirical conditional coverage in synthetic and real multivariate datasets. Practically, this yields data-dependent, computationally efficient conformal sets suitable for complex, high-dimensional prediction tasks with incomplete or evolving output information. Overall, the approach offers a flexible, scalable framework that enhances uncertainty quantification for multivariate predictions in conformal prediction settings.

Abstract

While achieving exact conditional coverage in conformal prediction is unattainable without making strong, untestable regularity assumptions, the promise of conformal prediction hinges on finding approximations to conditional guarantees that are realizable in practice. A promising direction for obtaining conditional dependence for conformal sets--in particular capturing heteroskedasticity--is through estimating the conditional density and conformalizing its level sets. Previous work in this vein has focused on nonconformity scores based on the empirical cumulative distribution function (CDF). Such scores are, however, computationally costly, typically requiring expensive sampling methods. To avoid the need for sampling, we observe that the CDF-based score reduces to a Mahalanobis distance in the case of Gaussian scores, yielding a closed-form expression that can be directly conformalized. Moreover, the use of a Gaussian-based score opens the door to a number of extensions of the basic conformal method; in particular, we show how to construct conformal sets with missing output values, refine conformal sets as partial information about becomes available, and construct conformal sets on transformations of the output space. Finally, empirical results indicate that our approach produces conformal sets that more closely approximate conditional coverage in multivariate settings compared to alternative methods.

Paper Structure

This paper contains 32 sections, 39 equations, 6 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Summary of contributions.
  3. Related work
  4. Background on conformal prediction with density estimation
  5. Notation.
  6. Gaussian Conformal Prediction
  7. Learning a Gaussian model
  8. Conformalizing the Gaussian density with multivariate standardized residuals
  9. Missing Outputs
  10. Method.
  11. Partially Revealed Outputs
  12. Method.
  13. Projection of the Output
  14. Method.
  15. Experiments
  16. ...and 17 more sections

Figures (6)

  • Figure 1: Illustrating a typical failure case of the $S_\text{NL}$ conformal score. The $-\hat{q}_\alpha$ likelihood threshold being the same for every $X$, when the variance of the estimated conditional distribution $\hat{p}(\cdot|X)$ is large, the conformal set produced may be empty (right plot). This is paradoxical as a larger variance indicates more uncertainty and should translate into larger conformal sets.
  • Figure 2: Illustrating the generalized CDF score $S_\text{HDP}$. With this score, a portion of the total mass of the estimated distribution $\hat{p}(\cdot|X)$ is conformalized to satisfy the required $1-\alpha$ coverage level on the calibration set. The size of the conformal sets constructed cannot be null, as it naturally follows the variance of the estimated conditional distribution, making this score robust to the failure case exposed for $S_\text{NL}$.
  • Figure 3: Conformal sets obtained with different score functions: $S_\text{Mah}$ (left), $S_\text{NL}$ (center) and $S_\text{ECM}$ (right). Data is generated from $Y \sim f(X) + T(X)B$, where $B$ is a standard normal vector and $T(\cdot)$ is a transformation that introduces heteroskedasticity in $X$ (see Appendix \ref{['app:synthetic:datageneration']} for details). We plot the results obtained for four different test points $X_i$ with different colors. Dark dots show the observed label $Y_i$, while light dots represent additional samples from $Y \mid X_i$ to illustrate the shape of the conditional distribution. The ellipsoids indicate the conformal set centered in $f_\theta(X_i)$ constructed using the score used with coverage parameter $1-\alpha$ set to $90\%$.
  • Figure 4: Conformal sets obtained for partially revealed outputs via the method described in Section \ref{['sec:partially:revealed']}. Data is generated from $Y \sim f(X) + T(X)B$, where $B$ is a vector from a standard normal (left) or exponential distribution and $T(\cdot)$ is a transformation that introduces heteroskedasticity in $X$ (see Appendix \ref{['app:synthetic:datageneration']} for details). We plot the results obtained for four different test point $X_i$ with different colors. Dark dots show the observed label $Y_i$, while light dots are more samples from $Y \mid X_i$ to illustrate the shape of the conditional distribution. Dashed ellipsoids represent the conformal sets constructed for the full output $(Y^1, Y^2)$ using the Mahalanobis score with coverage parameter $1-\alpha$ set to $90\%$. Full line intervals represent the confidence sets for $Y^2$ obtained with our updated conformal score when the value of $Y^1$ is revealed.
  • Figure 5: Comparing conformal sets obtained for projected outputs $\varphi(Y)$, by using the method presented in Section \ref{['sec:projection:output']} (full ellipsoids), or by taking the projection $\varphi(\tilde{C}_\alpha(X_i))$ of the Mahalanobis conformal set $\tilde{C}_\alpha(X_i)$ of non-transformed outputs (dashed ellipsoids). Data is generated from $Y \sim f(X) + T(X)B$, where $B$ is a standard normally (left) or exponentially (right) distributed vector and $T(\cdot)$ is a transformation that introduces heteroskedasticity in $X$ (see Appendix \ref{['app:synthetic:datageneration']} for details). $Y\in \mathbb{R}^3$, and we consider a projection $\varphi$ of $Y$ in $\mathbb{R}^2$. The coverage parameter $1 - \alpha$ is set to $90\%$. We plot the results obtained for four different test point $X_i$ with different colors. Dark dots show the observed transformed label $\varphi(Y_i)$, while light dots depict additional samples from $\varphi(Y) \mid X_i$ to illustrate the shape of the conditional distribution.
  • ...and 1 more figures

Theorems & Definitions (6)

  • Remark 1.1
  • Remark 2.1: Jointly convex formulation
  • Remark 2.2: Link with MSE
  • Remark 2.3: Mis-specified model
  • Remark 2.4: Generalization to elliptical distributions
  • Remark 3.1