Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Symmetric Aggregation of Conformity Scores for Efficient Uncertainty Sets

Nabil Alami, Jad Zakharia, Souhaib Ben Taieb

TL;DR

The work tackles uncertainty quantification for multiple predictive models within conformal prediction by introducing SACP, which transfers per-model nonconformity scores into e-values and then aggregates them with a symmetric function to produce a single, valid prediction set. Theoretical analysis establishes how aggregation choices preserve coverage and provides a worst-case bound on set length, while an efficiency-oriented variant, SACP++, selects the aggregation parameter to minimize average set length without sacrificing validity. Empirical results across OpenML regression and image classification benchmarks show SACP and SACP++ consistently achieve the nominal coverage and yield substantially shorter prediction sets than state-of-the-art baselines, with SACP++ often providing the best efficiency. The approach demonstrates the value of score-level symmetric aggregation and paves the way for data-driven aggregation strategies in conformal uncertainty quantification.

Abstract

Access to multiple predictive models trained for the same task, whether in regression or classification, is increasingly common in many applications. Aggregating their predictive uncertainties to produce reliable and efficient uncertainty quantification is therefore a critical but still underexplored challenge, especially within the framework of conformal prediction (CP). While CP methods can generate individual prediction sets from each model, combining them into a single, more informative set remains a challenging problem. To address this, we propose SACP (Symmetric Aggregated Conformal Prediction), a novel method that aggregates nonconformity scores from multiple predictors. SACP transforms these scores into e-values and combines them using any symmetric aggregation function. This flexible design enables a robust, data-driven framework for selecting aggregation strategies that yield sharper prediction sets. We also provide theoretical insights that help justify the validity and performance of the SACP approach. Extensive experiments on diverse datasets show that SACP consistently improves efficiency and often outperforms state-of-the-art model aggregation baselines.

Symmetric Aggregation of Conformity Scores for Efficient Uncertainty Sets

TL;DR

The work tackles uncertainty quantification for multiple predictive models within conformal prediction by introducing SACP, which transfers per-model nonconformity scores into e-values and then aggregates them with a symmetric function to produce a single, valid prediction set. Theoretical analysis establishes how aggregation choices preserve coverage and provides a worst-case bound on set length, while an efficiency-oriented variant, SACP++, selects the aggregation parameter to minimize average set length without sacrificing validity. Empirical results across OpenML regression and image classification benchmarks show SACP and SACP++ consistently achieve the nominal coverage and yield substantially shorter prediction sets than state-of-the-art baselines, with SACP++ often providing the best efficiency. The approach demonstrates the value of score-level symmetric aggregation and paves the way for data-driven aggregation strategies in conformal uncertainty quantification.

Abstract

Access to multiple predictive models trained for the same task, whether in regression or classification, is increasingly common in many applications. Aggregating their predictive uncertainties to produce reliable and efficient uncertainty quantification is therefore a critical but still underexplored challenge, especially within the framework of conformal prediction (CP). While CP methods can generate individual prediction sets from each model, combining them into a single, more informative set remains a challenging problem. To address this, we propose SACP (Symmetric Aggregated Conformal Prediction), a novel method that aggregates nonconformity scores from multiple predictors. SACP transforms these scores into e-values and combines them using any symmetric aggregation function. This flexible design enables a robust, data-driven framework for selecting aggregation strategies that yield sharper prediction sets. We also provide theoretical insights that help justify the validity and performance of the SACP approach. Extensive experiments on diverse datasets show that SACP consistently improves efficiency and often outperforms state-of-the-art model aggregation baselines.

Paper Structure

This paper contains 35 sections, 11 theorems, 59 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Conformal Prediction.
  4. CP aggregation methods.
  5. Setup Justification.
  6. Our SACP Method
  7. Theoretical analysis
  8. The aggregating function.
  9. Quantile aggregation
  10. An efficiency-oriented SACP method
  11. Related Work
  12. Experiments and Results
  13. Datasets.
  14. Baselines.
  15. Experimental setup.
  16. ...and 20 more sections

Key Result

Proposition 3.2

Consider exchangeable and positive random variables $\{S_i\}_{1 \leq i \leq n+1}$. Then, the random variables have expectation equal to one.

Figures (6)

  • Figure 1: Diagram illustrating key steps of SACP
  • Figure 2: Empirical coverage and average prediction set length per method across OpenML regression datasets.
  • Figure 3: Proof roadmap for bounding SACP prediction set
  • Figure 4: Cases of where $y_0,y_1$ lie in or out of $C^f_\alpha$.
  • Figure 5: An illustration of $U,L$ maps intersection points.
  • ...and 1 more figures

Theorems & Definitions (22)

  • Definition 3.1
  • Proposition 3.2
  • Theorem 3.3
  • Proposition 3.4
  • Proposition 3.5
  • Definition 3.6
  • Theorem 3.7: Worst-case bound
  • proof
  • proof
  • proof
  • ...and 12 more