Offline changepoint localization using a matrix of conformal p-values

TL;DR

The paper tackles offline changepoint localization by constructing a finite-sample valid confidence set for the changepoint index $\xi$ under exchangeability of the pre-change and post-change distributions. It introduces the MCP algorithm, which builds a matrix of conformal p-values using score functions and a conformal Neyman-Pearson lemma to produce left/right p-values and a unified test p-value $p_t$, enabling Neyman inversion to form $\mathcal{C}_{1-\alpha}$. The authors prove finite-sample coverage guarantees and demonstrate narrow confidence sets across MNIST, SST-2, HAR, and CIFAR-100, including challenging high-dimensional changes and even when the change is subtle. The approach supports learning the score function from data (e.g., via classifiers) and applies to diverse data modalities, offering a practical, assumption-light tool for reliable changepoint localization in complex settings.

Abstract

Changepoint localization is the problem of estimating the index at which a change occurred in the data generating distribution of an ordered list of data, or declaring that no change occurred. We present the broadly applicable MCP algorithm, which uses a matrix of conformal p-values to produce a confidence interval for a (single) changepoint under the mild assumption that the pre-change and post-change distributions are each exchangeable. We prove a novel conformal Neyman-Pearson lemma, motivating practical classifier-based choices for our conformal score function. Finally, we exemplify the MCP algorithm on a variety of synthetic and real-world datasets, including using black-box pre-trained classifiers to detect changes in sequences of images, text, and accelerometer data.

Figures (12)

• Figure 1: $\mathcal{H}_{0t} : \xi = t$ states that $(X_k)_{k=1}^t \sim \mathbb{P}_0$ and $(X_k)_{k=t+1}^n \sim \mathbb{P}_1$.
• Figure 2: Refinement of the p-values in the $t$th row of the matrix of conformal p-values (MCP) into the discrepancy statistics $W_t^{(0)}$ and $W_t^{(1)}$.
• Figure 3: Partial sample of MNIST digit change from "3" to "7" with a changepoint at $\xi = 400$.
• Figure 4: p-values for MNIST digit change at $\xi = 400$ using a pre-trained digit classifier. The dashed red line indicates the true changepoint, and the region on the horizontal axis where the p-values lie above the dotted green line ($\alpha = 0.05$) corresponds to our 95% confidence set. The point estimator is $\hat{\xi} = 403$, which is close to the true $\xi = 400$.
• Figure 5: p-values for SST-2 sentiment change at $\xi = 400$ using DistilBERT trained for sentiment analysis. The dashed red line indicates the true changepoint, and the region on the horizontal axis where the p-values lie above the dotted green line ($\alpha = 0.05$) corresponds to our 95% confidence set. The point estimators are a) $\hat{\xi} = 388$ and b) $\hat{\xi} = 373$, both of which are close to the true $\xi = 400$.

Theorems & Definitions (8)

• Theorem 3.1: Coverage guarantee (empirical test)
• Theorem 5.1: Conformal Neyman-Pearson lemma
• Theorem C.1: Asymptotic coverage guarantee (asymptotic KS test)
• Lemma E.1: Randomized probability integral transform
• proof : Proof of \ref{['lem:randomized-pit']}
• proof : Proof of \ref{['thm:coverage-empirical']}
• proof : Proof of \ref{['thm:coverage-asymp-ks']}
• proof : Proof of \ref{['thm:conformal-np']}