Distributed Conformal Prediction via Message Passing

TL;DR

This work addresses post-hoc calibration of pre-trained models in a fully decentralized setting where devices hold local calibration data and communicate only with neighbors over a graph. It proposes two conformal prediction schemes, Q-DCP and H-DCP, based on message passing: Q-DCP uses decentralized quantile regression via ADMM with smoothing and regularization to achieve linear convergence, while H-DCP leverages consensus on quantized score histograms to estimate the global distribution. The authors provide theoretical coverage guarantees for both methods and demonstrate trade-offs between hyperparameter tuning, communication overhead, and prediction set efficiency across various topologies, with code available for replication. The results support the viability of decentralized CP for reliable inference in privacy-preserving networks, such as healthcare IoT and distributed sensors, by balancing communication cost and coverage guarantees.

Abstract

Post-hoc calibration of pre-trained models is critical for ensuring reliable inference, especially in safety-critical domains such as healthcare. Conformal Prediction (CP) offers a robust post-hoc calibration framework, providing distribution-free statistical coverage guarantees for prediction sets by leveraging held-out datasets. In this work, we address a decentralized setting where each device has limited calibration data and can communicate only with its neighbors over an arbitrary graph topology. We propose two message-passing-based approaches for achieving reliable inference via CP: quantile-based distributed conformal prediction (Q-DCP) and histogram-based distributed conformal prediction (H-DCP). Q-DCP employs distributed quantile regression enhanced with tailored smoothing and regularization terms to accelerate convergence, while H-DCP uses a consensus-based histogram estimation approach. Through extensive experiments, we investigate the trade-offs between hyperparameter tuning requirements, communication overhead, coverage guarantees, and prediction set sizes across different network topologies. The code of our work is released on: https://github.com/HaifengWen/Distributed-Conformal-Prediction.

Figures (17)

• Figure 1: (a) This work studies a decentralized inference setting in which multiple devices share the same pre-trained model, and each device has a local calibration data set. (b) Given a common input, the devices aim at producing a prediction set that includes the true label with a user-defined probability $1-\alpha$. (c) We propose two message-passing schemes for conformal prediction (CP): Quantile-based distributed CP (Q-DCP) addresses the decentralized optimization of the pinball, or quantile, loss over the calibration scores; while histogram-based distributed CP (H-DCP) targets the consensus-based estimate of the histogram of the calibration scores.
• Figure 2: Coverage and normalized set size versus coverage level $1-\alpha$ for CP and Q-DCP when Assumption \ref{['assumption:regularization error']} is satisfied ($T=1500$ and $\epsilon_0=0.1$). The shaded error bars correspond to intervals covering 95% of the realized values.
• Figure 3: Coverage and normalized set size versus coverage level $1-\alpha$ for CP and Q-DCP when Assumption \ref{['assumption:regularization error']} is not satisfied ($T=1500$, $\epsilon_0=10^{-4}$, and $s_0=-8-20\alpha$).
• Figure 4: Coverage and normalized set size versus coverage level $1-\alpha$ for CP and H-DCP ($T=150$).
• Figure 5: Coverage and normalized set size versus the total per-device communication load (torus graph with $\alpha=0.1$, and Q-DCP with hyperparameter setting $1$ given by ($\epsilon_0=0.1$, $s_0$ being the average of the local score quantile) and hyperparameter setting 2 given by ($\epsilon_0=10^{-4}$, $s_0=-10$)).

Theorems & Definitions (4)

• Proposition 4.2
• Proposition 4.3: Theorem 1, shi2014linear
• Theorem 4.4: Coverage Guarantee for Q-DCP
• Theorem 5.2: Coverage Guarantee for H-DCP