Merging uncertainty sets via majority vote

TL;DR

The paper addresses combining multiple potentially dependent uncertainty sets into a single set that retains nearly the same coverage as the inputs. It introduces a majority-vote framework, yielding the merged set $\mathcal{C}^M$, with extensions to thresholds, randomization, weights, and sequential settings; it also shows that exchangeability can tighten guarantees via $\mathcal{C}^E$ and provides derandomization strategies for split-conformal and HulC-style procedures. The authors develop theoretical guarantees (nonasymptotic) and practical algorithms, including computation strategies for 1D intervals, and demonstrate improvements in conformal prediction both in simulations and on real data (Parkinson’s dataset). Overall, the work offers a simple, robust, black-box method to fuse multiple uncertainty sets while enabling derandomization and improved efficiency in modern uncertainty quantification tasks.

Abstract

Given $K$ uncertainty sets that are arbitrarily dependent -- for example, confidence intervals for an unknown parameter obtained with $K$ different estimators, or prediction sets obtained via conformal prediction based on $K$ different algorithms on shared data -- we address the question of how to efficiently combine them in a black-box manner to produce a single uncertainty set. We present a simple and broadly applicable majority vote procedure that produces a merged set with nearly the same error guarantee as the input sets. We then extend this core idea in a few ways: we show that weighted averaging can be a powerful way to incorporate prior information, and a simple randomization trick produces strictly smaller merged sets without altering the coverage guarantee. Further improvements can be obtained if the sets are exchangeable. We also show that many modern methods, like split conformal prediction, median of means, HulC and cross-fitted ``double machine learning'', can be effectively derandomized using these ideas.

Figures (8)

• Figure 1: Visual representation of the majority vote procedure when $\cap_{k=1}^K \mathcal{C}_k \neq \varnothing$.
• Figure 2: First column: MoM estimator obtained during various replications in a a single run of the procedure using data generated from the t-distribution (above) and the skew-t distribution (below). The dashed line is the true mean $\mu$. Second column: MoMoM estimator obtained during the replications; both series stabilize for $k>40$. Third column: average over 1000 runs of the absolute difference $|\hat{\mu}^\mathrm{MoMoM}_k-\hat{\mu}^\mathrm{MoMoM}_{k-1}|$ against $k$ (above) and average over 1000 runs of the absolute difference $|\hat{\mu}^\mathrm{MoMoM}_k-\mu|$ against $K$. Note that the third column is only available to us in the simulation, but plots like the second column can be constructed on the fly as $K$ increases, and it can be adaptively tracked and stopped, while retaining the same statistical guarantee at the stopped $K$.
• Figure 3: First two columns: example of a single run of the procedure using 210 observations generated from the t-distribution. Left: confidence intervals obtained for different random splits, Right: the merged sets. Last two columns: average over $5 000$ runs of the empirical coverage (left) and length (right) of $\mathcal{C}^M$ and $\mathcal{C}^E$ against $K$.
• Figure 4: Intervals obtained using different values of $\lambda$, $\mathcal{C}^M(x)$, $\mathcal{C}^R(x), \mathcal{C}^U(x)$ and $\mathcal{C}^\pi(x)$. For standardization, the value of $U$ in the randomized thresholds is set to $1/2$. The smallest set $\mathcal{C}^R$. Since $U=1/2$, the sets $\mathcal{C}^M$ and $\mathcal{C}^U$ coincides.
• Figure 5: Comparison between the simple majority vote ($\mathcal{C}^M$) and exchangeable majority vote ($\mathcal{C}^E$) with different thresholds $\tau$. The size of the $\mathcal{C}^E$ is significantly smaller than $\mathcal{C}^M$.

Theorems & Definitions (36)

• Theorem 2.1
• proof
• proof
• Remark 2.2
• Remark 2.3: When does majority vote overcover and when does it undercover?
• Theorem 2.4
• Theorem 2.5
• Lemma 2.6
• Theorem 2.7
• Theorem 2.8