Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Conformal Blindness: A Note on $A$-Cryptic change-points

Johan Hallberg Szabadváry

TL;DR

Conformal predictions rely on CTMs to test exchangeability by observing uniform post-change p-values, but the converse implication (uniformity implying no distribution shift) fails. The authors construct an explicit $A$-cryptic change-point using an oracle conformity measure $A(x,y)=f_{Y|X}(y|x)$ in a bivariate Gaussian setting, identifying a line along which mean shifts preserve the conformity-score distribution and keep $p$-values uniform. This yields conformal blindness: drastic shifts can go undetected by CTMs, while predictive validity and efficiency may remain intact for such shifts. The work highlights that the choice of conformity measure governs detectability of distribution shifts, suggesting robustness approaches (e.g., ensembles, joint p-values) and motivating further characterization of conditions under which $A$-cryptic pairs exist.

Abstract

Conformal Test Martingales (CTMs) are a standard method within the Conformal Prediction framework for testing the crucial assumption of data exchangeability by monitoring deviations from uniformity in the p-value sequence. Although exchangeability implies uniform p-values, the converse does not hold. This raises the question of whether a significant break in exchangeability can occur, such that the p-values remain uniform, rendering CTMs blind. We answer this affirmatively, demonstrating the phenomenon of \emph{conformal blindness}. Through explicit construction, for the theoretically ideal ``oracle'' conformity measure (given by the true conditional density), we demonstrate the possibility of an \emph{$A$-cryptic change-point} (where $A$ refers to the conformity measure). Using bivariate Gaussian distributions, we identify a line along which a change in the marginal means does not alter the distribution of the conformity scores, thereby producing perfectly uniform p-values. Simulations confirm that even a massive distribution shift can be perfectly cryptic to the CTM, highlighting a fundamental limitation and emphasising the critical role of the alignment of the conformity measure with potential shifts.

Conformal Blindness: A Note on $A$-Cryptic change-points

TL;DR

Conformal predictions rely on CTMs to test exchangeability by observing uniform post-change p-values, but the converse implication (uniformity implying no distribution shift) fails. The authors construct an explicit -cryptic change-point using an oracle conformity measure in a bivariate Gaussian setting, identifying a line along which mean shifts preserve the conformity-score distribution and keep -values uniform. This yields conformal blindness: drastic shifts can go undetected by CTMs, while predictive validity and efficiency may remain intact for such shifts. The work highlights that the choice of conformity measure governs detectability of distribution shifts, suggesting robustness approaches (e.g., ensembles, joint p-values) and motivating further characterization of conditions under which -cryptic pairs exist.

Abstract

Conformal Test Martingales (CTMs) are a standard method within the Conformal Prediction framework for testing the crucial assumption of data exchangeability by monitoring deviations from uniformity in the p-value sequence. Although exchangeability implies uniform p-values, the converse does not hold. This raises the question of whether a significant break in exchangeability can occur, such that the p-values remain uniform, rendering CTMs blind. We answer this affirmatively, demonstrating the phenomenon of \emph{conformal blindness}. Through explicit construction, for the theoretically ideal ``oracle'' conformity measure (given by the true conditional density), we demonstrate the possibility of an \emph{-cryptic change-point} (where refers to the conformity measure). Using bivariate Gaussian distributions, we identify a line along which a change in the marginal means does not alter the distribution of the conformity scores, thereby producing perfectly uniform p-values. Simulations confirm that even a massive distribution shift can be perfectly cryptic to the CTM, highlighting a fundamental limitation and emphasising the critical role of the alignment of the conformity measure with potential shifts.
Paper Structure (5 sections, 10 equations, 3 figures)

This paper contains 5 sections, 10 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Problem formulation
  3. Construction
  4. Illustration
  5. Discussion

Figures (3)

  • Figure 1: Detection of a non-cryptic change-point. (a) A shift in data distribution occurs. (b) The resulting p-value distribution becomes clearly non-uniform. (c) Consequently, the Conformal Test Martingale successfully grows, detecting the break in exchangeability.
  • Figure 2: Visual demonstration of a conformal-cryptic change-point. (a) A massive shift in the data distribution is visually apparent. (b) The resulting p-value histogram remains perfectly uniform. (c) Consequently, the Conformal Test Martingale fails to grow, remaining blind to the shift.
  • Figure 3: Visual demonstration of predictive efficiency. (a) A moderate shift in the data distribution results in clearly increasing interval lengths. (b) A massive $A$-cryptic shift results in no change in efficiency. Prediction intervals remain valid and efficient.

Theorems & Definitions (1)

  • Definition 1