Conformal Prediction Under Distribution Shift: A COVID-19 Natural Experiment

TL;DR

Conformal prediction guarantees are fragile under distribution shift, which the authors study with a COVID-19 induced shift across 8 supply-chain tasks using the rel-salt dataset. They show coverage degradation up to $86.7\%$ and that SHAP-based concentration predicts catastrophic failures with $\rho=0.714$ and $p=0.047$. Adaptive Conformal Inference fails under severe shift, while quarterly retraining improves coverage for vulnerable tasks by up to about $19$ percentage points, with robust tasks largely unaffected. A practitioner decision framework is proposed: monitor SHAP concentration before deployment, retrain quarterly if concentration is high ($>40\%$), and skip retraining for robust tasks; exploratory results indicate the mechanism depends on feature stability and extends to regression tasks. The work provides actionable guidance for deploying conformal prediction in non-stationary settings, emphasizing the trade-off between reliability and computational cost.

Abstract

Conformal prediction guarantees degrade under distribution shift. We study this using COVID-19 as a natural experiment across 8 supply chain tasks. Despite identical severe feature turnover (Jaccard approximately 0), coverage drops vary from 0% to 86.7%, spanning two orders of magnitude. Using SHapley Additive exPlanations (SHAP) analysis, we find catastrophic failures correlate with single-feature dependence (rho = 0.714, p = 0.047). Catastrophic tasks concentrate importance in one feature (4.5x increase), while robust tasks redistribute across many (10-20x). Quarterly retraining restores catastrophic task coverage from 22% to 41% (+19 pp, p = 0.04), but provides no benefit for robust tasks (99.8% coverage). Exploratory analysis of 4 additional tasks with moderate feature stability (Jaccard 0.13-0.86) reveals feature stability, not concentration, determines robustness, suggesting concentration effects apply specifically to severe shifts. We provide a decision framework: monitor SHAP concentration before deployment; retrain quarterly if vulnerable (>40% concentration); skip retraining if robust.

Figures (5)

• Figure 1: Main results: (A) Coverage degradation across tasks, (B) Drop magnitude with severity thresholds, (C) Complexity vs vulnerability scatter, (D) 2$\times$2 vulnerability taxonomy.
• Figure 2: Feature Importance Analysis reveals mechanism of catastrophic failure. (A) Catastrophic task: dominant feature SALESDOCUMENT explodes 4.5× while maintaining top rank. (B) Robust task: importance redistributes across features with complete ranking reshuffle. (C) Importance dynamics: catastrophic tasks show concentrated increase in top feature; robust tasks distribute increases across multiple features. (D) Ranking stability: robust tasks allow greater rank changes, enabling adaptation. Despite $\sim$0% Jaccard similarity, coverage drops differ by 700× due to feature importance dynamics.
• Figure 3: SHAP Concentration Predicts Coverage Degradation in Severe-Shift Scenarios. Scatter plot shows correlation between SHAP concentration and coverage drop for 8 tasks with complete feature turnover (Spearman $\rho=0.714$, $p=0.047$). Exploratory data from 4 tasks with moderate feature stability (Jaccard 0.13--0.86, shown in lighter colors) reveal a different mechanism where feature stability determines robustness. The 40% concentration threshold applies to severe-shift scenarios with Jaccard $\approx$0.
• Figure 4: Retraining restores coverage for catastrophic tasks. Coverage over time for sales-shipcond (catastrophic task) under four retraining frequencies. Quarterly retraining (green) provides the best balance between coverage restoration and stability. Monthly retraining (blue) shows high variance and occasional failures. No retraining (red) exhibits severe degradation post-COVID (month 6). Robust task (sales-office, not shown) maintains 99.8% coverage regardless of retraining.
• Figure 5: Extended experiments: (A) ACI fails under severe shift, (B) Placebo test shows COVID causes 10--100$\times$ more degradation than normal temporal drift, (C) rel-trial shows same pattern, (D) Feature overlap predicts failure ($r=-0.70$).