A Multimodal Deep Learning Framework for Predicting ICU Deterioration: Integrating ECG Waveforms with Clinical Data and Clinician Benchmarking

TL;DR

MDS ICU, a unified multimodal machine learning framework that fuses routinely collected data including demographics, biometrics, vital signs, laboratory values, ECG waveforms, surgical procedures, and medical device usage, demonstrates that multimodal AI can deliver clinically meaningful risk stratification across diverse ICU outcomes while augmenting rather than replacing clinical expertise.

Abstract

Artificial intelligence holds strong potential to support clinical decision making in intensive care units where timely and accurate risk assessment is critical. However, many existing models focus on isolated outcomes or limited data types, while clinicians integrate longitudinal history, real time physiology, and heterogeneous clinical information. To address this gap, we developed MDS ICU, a unified multimodal machine learning framework that fuses routinely collected data including demographics, biometrics, vital signs, laboratory values, ECG waveforms, surgical procedures, and medical device usage to provide continuous predictive support during ICU stays. Using 63001 samples from 27062 patients in MIMIC IV, we trained a deep learning architecture that combines structured state space S4 encoders for ECG waveforms with multilayer perceptron RealMLP encoders for tabular data to jointly predict 33 clinically relevant outcomes spanning mortality, organ dysfunction, medication needs, and acute deterioration. The model achieved strong discrimination with AUROCs of 0.90 for 24 hour mortality, 0.92 for sedative administration, 0.97 for invasive mechanical ventilation, and 0.93 for coagulation dysfunction. Calibration analysis showed close agreement between predicted and observed risks, with consistent gains from ECG waveform integration. Comparisons with clinicians and large language models showed that model predictions alone outperformed both, and that providing model outputs as decision support further improved their performance. These results demonstrate that multimodal AI can deliver clinically meaningful risk stratification across diverse ICU outcomes while augmenting rather than replacing clinical expertise, establishing a scalable foundation for precision critical care decision support.

Figures (8)

• Figure 1: Overview of the MDS-ICU clinical and predictive modeling workflow. For each 10-second, 12-lead ECG recorded during an ICU stay, a corresponding observation window is constructed, spanning from ICU admission up to the ECG acquisition time. Within this observation window, multimodal clinical data, including demographics, vital signs, laboratory measurements, and biometrics, are extracted to form the feature set. These inputs feed into a unified model trained to perform a wide range of prediction tasks, such as clinical deterioration, early warning scores, medication administrations, discharge diagnoses, discharges, and mortalities. The dataset integrates and harmonizes structured clinical records and waveform data from MIMIC-IV, MIMIC-IV-ECG, and MIMIC-IV-ECG-ICD, enabling a comprehensive and temporally aligned view of each ICU episode. Rigorous preprocessing ensures the clinical plausibility and quality of extracted features, establishing a robust foundation for training generalizable and clinically useful decision support systems.
• Figure 2: Calibration plots for a subset of representative labels, one per category: mortality (within stay), invasive mechanical ventilation (clinical deterioration), vasopressors (medications), and SOFA kidney (organ dysfunction). Probabilities were calibrated using isotonic regression on the validation set. Both models are overall well-calibrated, with S4+RealMLP showing improved alignment with observed outcomes, particularly in the higher probability ranges. Calibration was computed using a quantile strategy with 10 bins.
• Figure 3: Clinical benchmark plots for a subset of representative labels, one per category: mortality (within stay), invasive mechanical ventilation (clinical deterioration), vasopressors (medications), and SOFA kidney (organ dysfunction). Each plot contains the the main investigated model (S4+RealMLP) AUROC, as well as sensitivity and specificity of clinicians, GPT, and Claude for both benchmark A (clinician/LLM alone) and B (clinician/LLM+model).
• Figure 4: Overview of the multimodal architecture. The data flow proceeds from left to right. The top branch shows the time-series encoder based on structured state space models (S4), consisting of four S4 blocks followed by a pooling head. The bottom branch depicts the tabular encoder implemented using RealMLP, which applies robust quantile-based feature scaling, learnable feature re-scaling, and a stack of scaled linear layers (NTPLinear) with SELU activations. The outputs of both encoders are concatenated to form a joint multimodal representation used for prediction.
• Figure 5: Calibration figures for the mortality category.