Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adversarial Drift-Aware Predictive Transfer: Toward Durable Clinical AI

Xin Xiong, Zijian Guo, Haobo Zhu, Chuan Hong, Jordan W Smoller, Tianxi Cai, Molei Liu

TL;DR

Adversarial Drift-Aware Predictive Transfer (ADAPT), a novel framework designed to confer durability against temporal drift with minimal retraining, offers a scalable pathway for sustaining reliable AI in high-stakes healthcare environments.

Abstract

Clinical AI systems frequently suffer performance decay post-deployment due to temporal data shifts, such as evolving populations, diagnostic coding updates (e.g., ICD-9 to ICD-10), and systemic shocks like the COVID-19 pandemic. Addressing this ``aging'' effect via frequent retraining is often impractical due to computational costs and privacy constraints. To overcome these hurdles, we introduce Adversarial Drift-Aware Predictive Transfer (ADAPT), a novel framework designed to confer durability against temporal drift with minimal retraining. ADAPT innovatively constructs an uncertainty set of plausible future models by combining historical source models and limited current data. By optimizing worst-case performance over this set, it balances current accuracy with robustness against degradation due to future drifts. Crucially, ADAPT requires only summary-level model estimators from historical periods, preserving data privacy and ensuring operational simplicity. Validated on longitudinal suicide risk prediction using electronic health records from Mass General Brigham (2005--2021) and Duke University Health Systems, ADAPT demonstrated superior stability across coding transitions and pandemic-induced shifts. By minimizing annual performance decay without labeling or retraining future data, ADAPT offers a scalable pathway for sustaining reliable AI in high-stakes healthcare environments.

Adversarial Drift-Aware Predictive Transfer: Toward Durable Clinical AI

TL;DR

Adversarial Drift-Aware Predictive Transfer (ADAPT), a novel framework designed to confer durability against temporal drift with minimal retraining, offers a scalable pathway for sustaining reliable AI in high-stakes healthcare environments.

Abstract

Clinical AI systems frequently suffer performance decay post-deployment due to temporal data shifts, such as evolving populations, diagnostic coding updates (e.g., ICD-9 to ICD-10), and systemic shocks like the COVID-19 pandemic. Addressing this ``aging'' effect via frequent retraining is often impractical due to computational costs and privacy constraints. To overcome these hurdles, we introduce Adversarial Drift-Aware Predictive Transfer (ADAPT), a novel framework designed to confer durability against temporal drift with minimal retraining. ADAPT innovatively constructs an uncertainty set of plausible future models by combining historical source models and limited current data. By optimizing worst-case performance over this set, it balances current accuracy with robustness against degradation due to future drifts. Crucially, ADAPT requires only summary-level model estimators from historical periods, preserving data privacy and ensuring operational simplicity. Validated on longitudinal suicide risk prediction using electronic health records from Mass General Brigham (2005--2021) and Duke University Health Systems, ADAPT demonstrated superior stability across coding transitions and pandemic-induced shifts. By minimizing annual performance decay without labeling or retraining future data, ADAPT offers a scalable pathway for sustaining reliable AI in high-stakes healthcare environments.
Paper Structure (36 sections, 9 equations, 7 figures, 1 table)

This paper contains 36 sections, 9 equations, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Method
  3. The ADAPT Framework
  4. Overview.
  5. Step 1: Capturing Historical and Current Signals.
  6. Step 2: Defining the Horizon of Plausible Futures.
  7. Step 3: Adversarial Optimization for Robustness.
  8. Summary of Advantages.
  9. Validation Studies
  10. Validation Using Synthetic Data
  11. Data Generation.
  12. Simulation Configurations.
  13. Benchmarks and Evaluation Metric.
  14. Real World Case Study of Suicide Risk Prediction Model Degradation
  15. Description of the Study Cohorts.
  16. ...and 21 more sections

Figures (7)

  • Figure 1: (Left panel) Three-step ADAPT algorithm: (1) fit period-specific models for the current target year and prior years to obtain a set of candidate models; (2) use the current and historical models to characterize the range of plausible futures by specifying an uncertainty set over candidate models that likely represents future drift; and (3) solve an adversarial robustness optimization problem over this region to produce the ADAPT model. (Right panel) Expanded view of Step 3 with the degradation motivation: model performance decays as calendar time moves away from the training period (older $\boldsymbol{\beta}_t$ degrade on later data), and ADAPT selects $\boldsymbol{\beta}_{\mathrm{ADAPT}}$ as a present-time robust solution anchored at $\boldsymbol{\beta}_0$ that hedges against plausible near-future drift within the uncertainty set $C(\tau)$, improving stability against degradation.
  • Figure 2: AUC as the current-to-historical data ratio increases. Left: validation data come from the same period as the the training data; right: validation data come from periods after the training data.
  • Figure 3: AUC as perturbation level increases. Time window for the training target and validation data is split into two groups: before perturbation ($t\leq 7$) and after perturbtion ($t\geq 8$)
  • Figure 4: AUC when the training year and the validation year are the same.
  • Figure 5: AUC when the validation data comes from 2020 while models are trained for a target period (shown on x-axis) prior to 2020.
  • ...and 2 more figures