Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimizing In-Context Learning for Efficient Full Conformal Prediction

Weicao Deng, Sangwoo Park, Min Li, Osvaldo Simeone

TL;DR

This work tackles reliable uncertainty quantification with Conformal Prediction (CP), addressing the inefficiency of Split CP (SCP) and the retraining cost of Full CP (FCP). It introduces Enhanced ICL-based FCP (E-ICL+FCP), a permutation-invariant Transformer trained with a CP-aware loss to simulate $|\,\mathcal{Y}\,|$ retrained models in parallel, preserving the CP coverage $P(y_{n+1}\in \mathcal{C}(\mathbf{x}_{n+1}))\ge 1-\alpha$ while reducing computation. The method uses a differentiable meta-training objective with soft quantile $\hat{Q}_{1-\alpha}(\cdot)$ and soft indicator to minimize the expected predictive-set size, without actual gradient-based retraining at inference. Empirical results on synthetic tasks (symbol demodulation) and CIFAR-FS show that E-ICL+FCP achieves smaller predictive sets and favorable efficiency-coverage trade-offs compared to SCP and FCP baselines, including prior ICL-based CP methods. This work thus advances practical full conformal prediction by integrating CP-aware meta-learning with in-context learning.

Abstract

Reliable uncertainty quantification is critical for trustworthy AI. Conformal Prediction (CP) provides prediction sets with distribution-free coverage guarantees, but its two main variants face complementary limitations. Split CP (SCP) suffers from data inefficiency due to dataset partitioning, while full CP (FCP) improves data efficiency at the cost of prohibitive retraining complexity. Recent approaches based on meta-learning or in-context learning (ICL) partially mitigate these drawbacks. However, they rely on training procedures not specifically tailored to CP, which may yield large prediction sets. We introduce an efficient FCP framework, termed enhanced ICL-based FCP (E-ICL+FCP), which employs a permutation-invariant Transformer-based ICL model trained with a CP-aware loss. By simulating the multiple retrained models required by FCP without actual retraining, E-ICL+FCP preserves coverage while markedly reducing both inefficiency and computational overhead. Experiments on synthetic and real tasks demonstrate that E-ICL+FCP attains superior efficiency-coverage trade-offs compared to existing SCP and FCP baselines.

Optimizing In-Context Learning for Efficient Full Conformal Prediction

TL;DR

This work tackles reliable uncertainty quantification with Conformal Prediction (CP), addressing the inefficiency of Split CP (SCP) and the retraining cost of Full CP (FCP). It introduces Enhanced ICL-based FCP (E-ICL+FCP), a permutation-invariant Transformer trained with a CP-aware loss to simulate retrained models in parallel, preserving the CP coverage while reducing computation. The method uses a differentiable meta-training objective with soft quantile and soft indicator to minimize the expected predictive-set size, without actual gradient-based retraining at inference. Empirical results on synthetic tasks (symbol demodulation) and CIFAR-FS show that E-ICL+FCP achieves smaller predictive sets and favorable efficiency-coverage trade-offs compared to SCP and FCP baselines, including prior ICL-based CP methods. This work thus advances practical full conformal prediction by integrating CP-aware meta-learning with in-context learning.

Abstract

Reliable uncertainty quantification is critical for trustworthy AI. Conformal Prediction (CP) provides prediction sets with distribution-free coverage guarantees, but its two main variants face complementary limitations. Split CP (SCP) suffers from data inefficiency due to dataset partitioning, while full CP (FCP) improves data efficiency at the cost of prohibitive retraining complexity. Recent approaches based on meta-learning or in-context learning (ICL) partially mitigate these drawbacks. However, they rely on training procedures not specifically tailored to CP, which may yield large prediction sets. We introduce an efficient FCP framework, termed enhanced ICL-based FCP (E-ICL+FCP), which employs a permutation-invariant Transformer-based ICL model trained with a CP-aware loss. By simulating the multiple retrained models required by FCP without actual retraining, E-ICL+FCP preserves coverage while markedly reducing both inefficiency and computational overhead. Experiments on synthetic and real tasks demonstrate that E-ICL+FCP attains superior efficiency-coverage trade-offs compared to existing SCP and FCP baselines.

Paper Structure

This paper contains 15 sections, 14 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Split Conformal Prediction
  4. Full Conformal Prediction
  5. In-Context Learning
  6. Optimizing ICL for Efficient Full Conformal Prediction
  7. ICL-based FCP
  8. Enhanced ICL-Based FCP
  9. Meta-Training Setting
  10. Soft Quantile, Soft Indicator, and Differentiable Meta-Training Function
  11. Experiments
  12. Benchmarks
  13. Symbol Demodulation
  14. CIFAR-FS Image Classification
  15. Conclusions

Figures (3)

  • Figure 1: Illustration of the proposed enhanced ICL-based FCP (E-ICL+FCP). E-ICL+FCP employs a permutation-invariant Transformer-based ICL model trained with a CP-aware loss. The model simulates in parallel the retrained models $\{\hat{p}^y(\cdot|\mathbf{x})\}_{y\in \mathcal{Y}}$ required by FCP without implementing any actual retraining as in conventional FCP.
  • Figure 2: Coverage probability and predictive set size achieved by different schemes on the QPSK demodulation tasks ($\textcolor{red}{\bigstar}$ marks the mean, $\alpha=0.1$).
  • Figure 3: Coverage probability and predictive set size achieved by ICL-based schemes on the image classification tasks ($\textcolor{red}{\bigstar}$ marks the mean, $\alpha=0.1$).