Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Asymptotics for conformal inference

Ulysse Gazin

TL;DR

The paper resolves the exact asymptotic distribution of conformal inference error processes when both calibration and test samples grow, revealing a Brownian-bridge limit after scaling by $\sqrt{\tau_{n,m}}$ with $\tau_{n,m}=nm/(n+m)$ and connecting the limit to the Kolmogorov distribution. It shows how distribution shift inflates variance and shifts the mean via the function $G$, and how weighted conformal p-values—especially the oracle weights—can restore nominal behavior in the limit. The authors extend the results to novelty detection, deriving full asymptotic distributions for FDP/TDP under BH, including weighted and non-oracle scenarios. These results provide practical, asymptotically-valid quantiles for conformal inference in large-sample regimes and quantify the impact of shifts and weighting on inference accuracy. The framework lays a foundation for future work on estimated weights and broader dependency structures in conformal procedures.

Abstract

Conformal inference is a versatile tool for building prediction sets in regression or classification. We study the false coverage proportion (FCP) in a transductive setting with a calibration sample of $n$ points and a test sample of $m$ points. We identify the exact, distribution-free, asymptotic distribution of the FCP when both $n$ and $m$ tend to infinity. This shows in particular that FCP control can be achieved by using the well-known Kolmogorov distribution, and puts forward that the asymptotic variance is decreasing in the ratio $n/m$. We then provide a number of extensions by considering the novelty detection problem, weighted conformal inference and distribution shift between the calibration sample and the test sample. In particular, our asymptotic results allow to accurately quantify the asymptotic behavior of the errors when weighted conformal inference is used.

Asymptotics for conformal inference

TL;DR

The paper resolves the exact asymptotic distribution of conformal inference error processes when both calibration and test samples grow, revealing a Brownian-bridge limit after scaling by with and connecting the limit to the Kolmogorov distribution. It shows how distribution shift inflates variance and shifts the mean via the function , and how weighted conformal p-values—especially the oracle weights—can restore nominal behavior in the limit. The authors extend the results to novelty detection, deriving full asymptotic distributions for FDP/TDP under BH, including weighted and non-oracle scenarios. These results provide practical, asymptotically-valid quantiles for conformal inference in large-sample regimes and quantify the impact of shifts and weighting on inference accuracy. The framework lays a foundation for future work on estimated weights and broader dependency structures in conformal procedures.

Abstract

Conformal inference is a versatile tool for building prediction sets in regression or classification. We study the false coverage proportion (FCP) in a transductive setting with a calibration sample of points and a test sample of points. We identify the exact, distribution-free, asymptotic distribution of the FCP when both and tend to infinity. This shows in particular that FCP control can be achieved by using the well-known Kolmogorov distribution, and puts forward that the asymptotic variance is decreasing in the ratio . We then provide a number of extensions by considering the novelty detection problem, weighted conformal inference and distribution shift between the calibration sample and the test sample. In particular, our asymptotic results allow to accurately quantify the asymptotic behavior of the errors when weighted conformal inference is used.
Paper Structure (42 sections, 30 theorems, 106 equations, 2 figures)

This paper contains 42 sections, 30 theorems, 106 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Aim and contributions
  4. Relation to previous work
  5. Preliminaries
  6. Prediction setting
  7. Novelty detection setting
  8. Spaces for process convergence
  9. Asymptotics for conformal prediction
  10. Main result
  11. Weighted case
  12. Asymptotics for novelty detection
  13. Additional notation and assumptions
  14. Main results
  15. Proofs
  16. ...and 27 more sections

Key Result

Theorem 3.1

Under Assumption as:CP with $F_{\tiny \hbox{cal}}=F_{\tiny \hbox{test}}$, we have where $\tau_{n,m}$ is defined by eq:taunm and $\mathbb{U}$ is a standard Brownian bridge.

Figures (2)

  • Figure 1: Comparison of the $(1-\delta)$-quantile of different approximations of the distribution of $\| \mathrm{FCP}_m^{(n)}- I_n\|_\infty$ in \ref{['equ-diffinfty']} for different values of $n,m$ and $\delta$. The approximations include Monte-Carlo ($1000$ replications), DKW gazin2023transductive and the new asymptotic one (see text).
  • Figure 2: Plot of the asymptotic confidence interval for the $\mathrm{FCP}$ of the weighted conformal method (at level $80\%$) obtained in Theorem \ref{['thr:cvAlterWeight']} versus an error parameter $\Delta$. The calibration sample and the test sample are distributed according to the exponential distribution with mean $1$ and $1/3$, respectively. The weight function used in the conformal method is $w_\Delta(x)=\exp(-(2+\Delta)x){\mathds{1}_{x>0}}$, which corresponds to the oracle choice if $\Delta=0$ (for which the asymptotic average of $\mathrm{FCP}(\alpha)$ is equal to $\alpha$) and deviates from it if $\Delta\neq 0$.

Theorems & Definitions (40)

  • Theorem 3.1
  • Remark 3.1
  • Theorem 3.2
  • Theorem 3.3
  • Theorem 3.4
  • Theorem 4.1
  • Theorem 4.2
  • Proposition 5.1
  • Proposition 5.2
  • Proposition 5.3
  • ...and 30 more