Multi-model Online Conformal Prediction with Graph-Structured Feedback

TL;DR

This work tackles the challenge of reliable uncertainty quantification under distribution shift by developing graph-guided, online multi-model conformal prediction. The proposed GMOCP algorithm dynamically prunes candidate models via a time-varying bipartite graph and selects a single model from a subset to build prediction sets with guaranteed coverage. An enhanced version, EGMOCP, integrates prediction-set length into the model-weight updates to further reduce set sizes while preserving validity, with theoretical guarantees of sublinear regret. Empirical results on CIFAR-derived datasets, TinyImageNet-C, and synthetic shifts demonstrate smaller prediction sets and lower computation compared with baselines, highlighting practical gains for real-time uncertainty quantification in non-stationary environments.

Abstract

Online conformal prediction has demonstrated its capability to construct a prediction set for each incoming data point that covers the true label with a predetermined probability. To cope with potential distribution shift, multi-model online conformal prediction has been introduced to select and leverage different models from a preselected candidate set. Along with the improved flexibility, the choice of the preselected set also brings challenges. A candidate set that includes a large number of models may increase the computational complexity. In addition, the inclusion of irrelevant models with poor performance may negatively impact the performance and lead to unnecessarily large prediction sets. To address these challenges, we propose a novel multi-model online conformal prediction algorithm that identifies a subset of effective models at each time step by collecting feedback from a bipartite graph, which is refined upon receiving new data. A model is then selected from this subset to construct the prediction set, resulting in reduced computational complexity and smaller prediction sets. Additionally, we demonstrate that using prediction set size as feedback, alongside model loss, can significantly improve efficiency by constructing smaller prediction sets while still satisfying the required coverage guarantee. The proposed algorithms are proven to ensure valid coverage and achieve sublinear regret. Experiments on real and synthetic datasets validate that the proposed methods construct smaller prediction sets and outperform existing multi-model online conformal prediction approaches.

Figures (3)

• Figure 1: (left): An illustrative example of the generated bipartite graph $G_t$ with $M$ learning models and $J=2$ selective nodes. (right) The selective node $v_2^{(s)}$ is chosen, and the subset $S_t$ includes all learning models connected to the selected node, highlighted by red edges.
• Figure 2: Evaluation of prediction sets constructed by 3 training configurations of DenseNet121 under $N = 5$ and $J = 2$ over 6000 timesteps. The top plot shows the size of the prediction sets, while the bottom plot shows the corresponding model weights $w_t^m$ over time. DenseNet121-120D consistently receives the highest weight, indicating a higher likelihood of being selected. Moreover, models with better performance (e.g., DenseNet121-120D) create significantly smaller prediction sets.
• Figure 3: Evaluation of prediction set sizes constructed by multi-model methods over 6000 timesteps. On average, EGMOCP produces smaller prediction sets compared to all other benchmarks. The average prediction set sizes for COMA, MOCP, GMOCP, and EGMOCP across the 6000 timesteps are 7.19, 15.26, 11.00, and 6.06, respectively.

Theorems & Definitions (10)

• Theorem 1
• Theorem 2
• Theorem 3
• Theorem 4
• Lemma 1
• Lemma 2
• Lemma 3
• Lemma 4
• Lemma 5
• Lemma 6