Stable Prediction of Adverse Events in Medical Time-Series Data

TL;DR

This work addresses the deployment challenge of early event prediction (EEP) in clinical time-series by introducing CAREBench, a multi-modal benchmark that jointly evaluates predictive accuracy and temporal stability of risk trajectories. It formalizes a stability metric via a local Lipschitz-based measure and applies it across six tasks drawn from three public datasets (EHRShot, MC-MED, MIMIC-IV) spanning EHR, waveform, and text modalities. The study comprehensively compares classical models, deep sequence models, and zero-shot LLMs, revealing pronounced trade-offs: conventional methods tend to be stable and reliable, while LLMs can achieve higher accuracy but at the cost of stability and interpretability, highlighting the need for evidence-aligned, stable risk trajectories in real-world monitoring. Overall, CAREBench provides a framework to foster development of clinically deployable EEP methods that balance discriminative performance with trajectory stability to earn clinician trust.

Abstract

Early event prediction (EEP) systems continuously estimate a patient's imminent risk to support clinical decision-making. For bedside trust, risk trajectories must be accurate and temporally stable, shifting only with new, relevant evidence. However, current benchmarks (a) ignore stability of risk scores and (b) evaluate mainly on tabular inputs, leaving trajectory behavior untested. To address this gap, we introduce CAREBench, an EEP benchmark that evaluates deployability using multi-modal inputs-tabular EHR, ECG waveforms, and clinical text-and assesses temporal stability alongside predictive accuracy. We propose a stability metric that quantifies short-term variability in per-patient risk and penalizes abrupt oscillations based on local-Lipschitz constants. CAREBench spans six prediction tasks such as sepsis onset and compares classical learners, deep sequence models, and zero-shot LLMs. Across tasks, existing methods, especially LLMs, struggle to jointly optimize accuracy and stability, with notably poor recall at high-precision operating points. These results highlight the need for models that produce evidence-aligned, stable trajectories to earn clinician trust in continuous monitoring settings. (Code: https://github.com/SeewonChoi/CAREBench.)

Figures (3)

• Figure 1: Example medical record of a patient and corresponding risk scores produced by a model at different timestamps. Top: The input $x$ is $a$ hours of sparse time-series data consisting of tabular (blue), waveform (red), and text (orange) modalities, and output $y$ is a binary indicator of sepsis onset in the next $h$ hours. Bottom: Risk curves over time for patient $x$, for two models, shown in green (ideal) and red (inferior). Intuitively, the green predictions are more stable than red since the gaps between nearby points are smaller. Our stability measure quantifies the degree of fluctuations over time as the average local Lipschitz constant in short windows of time.
• Figure 2: Risk score of a patient over the 4 hour horizon for sepsis onset prediction, plotted for Mamba, GPT-2, GPT2$_\text{AR}$, and Qwen3-32B models. The true label is False.
• Figure 3: Example of an LLM prompt-response pair for early event prediction of sepsis onset on the MC-MED dataset.