CAOS: Conformal Aggregation of One-Shot Predictors

TL;DR

CAOS introduces a data-efficient conformal framework for one-shot prediction by adaptively aggregating multiple reference-induced predictors and calibrating with a leave-one-out scheme. It achieves exact finite-sample marginal coverage even when conformity scores are non-exchangeable, via a monotonicity-based reduction to full conformal prediction. Empirically, CAOS yields substantially smaller, reliable prediction sets than split conformal baselines in both vision (facial landmarking) and language (RAFT one-shot text classification) tasks, highlighting improved uncertainty quantification under scarce labeled data. The approach offers practical gains for rapid adaptation of foundation models with principled uncertainty in low-data regimes.

Abstract

One-shot prediction enables rapid adaptation of pretrained foundation models to new tasks using only one labeled example, but lacks principled uncertainty quantification. While conformal prediction provides finite-sample coverage guarantees, standard split conformal methods are inefficient in the one-shot setting due to data splitting and reliance on a single predictor. We propose Conformal Aggregation of One-Shot Predictors (CAOS), a conformal framework that adaptively aggregates multiple one-shot predictors and uses a leave-one-out calibration scheme to fully exploit scarce labeled data. Despite violating classical exchangeability assumptions, we prove that CAOS achieves valid marginal coverage using a monotonicity-based argument. Experiments on one-shot facial landmarking and RAFT text classification tasks show that CAOS produces substantially smaller prediction sets than split conformal baselines while maintaining reliable coverage.

Figures (3)

• Figure 1: Prediction set size per landmark for exemplar-based facial landmarking. Each point corresponds to a landmark. CAOS yields smaller prediction sets than SCOS Avg. and SCOS Best, approaching oracle efficiency.
• Figure 2: Conformal prediction set sizes for one-shot facial landmarking for (top) and (bottom) on three test images. Landmarks are colored by prediction set size. Smaller sets (green) correspond to lower predictive uncertainty and are preferable.
• Figure 3: Comparison of (above) and (below) in terms of average prediction set size (x-axis, the smaller the better) and empirical coverage (color) for one-shot prediction with Llama2-7B on tasks from the RAFT dataset at $\alpha=0.1$. a) For 4/9 tasks, both and hit target coverage. b) For 3/9 tasks only hits target coverage. c) For 2/9 tasks both miss. consistently achieves smaller prediction sets (in 8/9 tasks).

Theorems & Definitions (10)

• Theorem 4.2: Finite-sample coverage of
• Lemma 5.1: Symmetry of $\tilde{s}_{\mathrm{caos}}$
• proof
• Lemma 5.2: Monotonicity of the score
• proof
• Lemma 5.3: Comparison of and full scores
• proof
• Corollary 5.4: Set inclusion
• proof
• proof : Proof of \ref{['thm:caos-coverage']}