Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When Your Model Stops Working: Anytime-Valid Calibration Monitoring

Tristan Farran

Abstract

Practitioners monitoring deployed probabilistic models face a fundamental trap: any fixed-sample test applied repeatedly over an unbounded stream will eventually raise a false alarm, even when the model remains perfectly stable. Existing methods typically lack formal error guarantees, conflate alarm time with changepoint location, and monitor indirect signals that do not fully characterize calibration. We present PITMonitor, an anytime-valid calibration-specific monitor that detects distributional shifts in probability integral transforms via a mixture e-process, providing Type I error control over an unbounded monitoring horizon as well as Bayesian changepoint estimation. On river's FriedmanDrift benchmark, PITMonitor achieves detection rates competitive with the strongest baselines across all three scenarios, although detection delay is substantially longer under local drift.

When Your Model Stops Working: Anytime-Valid Calibration Monitoring

Abstract

Practitioners monitoring deployed probabilistic models face a fundamental trap: any fixed-sample test applied repeatedly over an unbounded stream will eventually raise a false alarm, even when the model remains perfectly stable. Existing methods typically lack formal error guarantees, conflate alarm time with changepoint location, and monitor indirect signals that do not fully characterize calibration. We present PITMonitor, an anytime-valid calibration-specific monitor that detects distributional shifts in probability integral transforms via a mixture e-process, providing Type I error control over an unbounded monitoring horizon as well as Bayesian changepoint estimation. On river's FriedmanDrift benchmark, PITMonitor achieves detection rates competitive with the strongest baselines across all three scenarios, although detection delay is substantially longer under local drift.
Paper Structure (32 sections, 5 theorems, 24 equations, 4 figures, 1 table)

This paper contains 32 sections, 5 theorems, 24 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Probability Integral Transforms
  5. Conformal P-values
  6. E-values
  7. Method
  8. E-values via Density Betting
  9. The Mixture E-process
  10. Type I Error Control
  11. Changepoint Estimation
  12. Experiments
  13. Setup
  14. Scenarios.
  15. Stream layout.
  16. ...and 17 more sections

Key Result

Proposition 1

This result holds more generally under exchangeability of $(U_t)$; we assume i.i.d. throughout for simplicity. If $U_1, \ldots, U_t \overset{i.i.d.}{\sim} F$ with $F$ continuous,If $F$ is discretized or rounded, randomized tie-breaking can be used to preserve uniformity. then $R_t \sim \mathrm{Unif}

Figures (4)

  • Figure 1: TPR and FPR across all detectors and drift scenarios. The dashed line marks the nominal $\alpha = 0.05$.
  • Figure 2: Single-run PITMonitor trace. (a) Predicted vs. actual values. (b) PIT stream with a rolling mean. (c) Mixture e-process on a log scale, with a dashed line marking the threshold $1/\alpha$. (d) Pre-shift and post-shift PIT histograms.
  • Figure 3: Detection delay distributions. White bars mark the median; colored text indicates sample count.
  • Figure 4: Distribution of PITMonitor changepoint estimation error $|\hat{\tau} - \tau|$ across true-positive trials.

Theorems & Definitions (10)

  • Proposition 1: Rank Uniformity Under $H_0$
  • proof
  • Proposition 2: Density Betting Yields Valid E-values
  • proof
  • Proposition 3: Finite-Time Mean Gain
  • proof
  • Proposition 4: Efficient Recursion
  • proof
  • Theorem 1: Anytime-Valid False Alarm Control
  • proof