Conformal Prediction for Ensembles: Improving Efficiency via Score-Based Aggregation

TL;DR

This work tackles distribution-free uncertainty quantification for ensembles by extending conformal prediction to multivariate score space. It introduces Conformal Score Aggregation (CSA), which constructs a data-driven, convex, score-space quantile envelope using a nested family of score-frontier sets and a two-stage calibration to preserve exchangeability. CSA yields more informative prediction regions than naive aggregation while maintaining formal coverage, demonstrated across ImageNet classification, OpenML regression, and a predict-then-optimize traffic routing task. The approach offers a practical, scalable framework for uncertainty estimation in multi-modal ensembles with downstream decision-making implications.

Abstract

Distribution-free uncertainty estimation for ensemble methods is increasingly desirable due to the widening deployment of multi-modal black-box predictive models. Conformal prediction is one approach that avoids such distributional assumptions. Methods for conformal aggregation have in turn been proposed for ensembled prediction, where the prediction regions of individual models are merged as to retain coverage guarantees while minimizing conservatism. Merging the prediction regions directly, however, sacrifices structures present in the conformal scores that can further reduce conservatism. We, therefore, propose a novel framework that extends the standard scalar formulation of a score function to a multivariate score that produces more efficient prediction regions. We then demonstrate that such a framework can be efficiently leveraged in both classification and predict-then-optimize regression settings downstream and empirically show the advantage over alternate conformal aggregation methods.

Figures (10)

• Figure 1: CSA provides a principled extension to the standard conformal prediction pipeline by leveraging ideas from higher-dimensional quantile regression to define quantile envelopes $\widehat{\mathcal{Q}}$ instead of scalar quantiles $\widehat{q}$. It does so by evaluating a collection of score functions (here $s_1$ and $s_2$) over the calibration dataset to define $\mathcal{S}$, finding quantiles $\{\widehat{q}_{m}\}$ over a set of projection directions $\{u_{m}\}$, and taking $\widehat{\mathcal{Q}}$ to be the intersection of the resulting half-planes $H(u_{m},\widehat{q}_{m})$. These quantile envelopes result in more informative prediction regions that can be used in downstream tasks.
• Figure 2: The calibration score evaluations are first split between those used to define the pre-ordering (green) $\mathcal{S}_C^{(1)}$ and those used to define the final multivariate quantile (red) $\mathcal{S}_C^{(2)}$.
• Figure 3: The pre-ordering points are projected across a number of directions, after which the $\beta$ quantile is used to define a direction quantile. This defines a half-plane of points that are in the region (blue) and those outside (red).
• Figure 4: We use the intersection of hyperplanes to define the quantile envelope, seeking $\beta^{*}$ that achieves the desired coverage.
• Figure 5: Using the quantile envelope, the family of nested sets $\mathcal{A}_t$ is defined, in turn defining a partial ordering over $\mathbb{R}^{K}$.

Theorems & Definitions (3)

• Theorem 3.1
• Theorem B.1
• proof