PULSE-ICU: A Pretrained Unified Long-Sequence Encoder for Multi-task Prediction in Intensive Care Units

TL;DR

ICU data are highly irregular and heterogeneous, challenging generalizable prediction. PULSE-ICU introduces a self-supervised foundation approach that encodes ICU events with a unified, multi-embedding representation and a Longformer encoder to model long trajectories without resampling. Pretrained on MIMIC-IV with masked event and value prediction, it is fine-tuned across 18 tasks and demonstrates strong within-domain performance and robust cross-domain transfer to HiRID, eICU, and PhysioNet 2012 with minimal fine-tuning. The results show data-efficient adaptation under limited-variable conditions and highlight the potential of foundation-style ICU representations for scalable, cross-institution decision support.

Abstract

Intensive care unit (ICU) data are highly irregular, heterogeneous, and temporally fragmented, posing challenges for generalizable clinical prediction. We present PULSE-ICU, a self-supervised foundation model that learns event-level ICU representations from large-scale EHR sequences without resampling or manual feature engineering. A unified embedding module encodes event identity, continuous values, units, and temporal attributes, while a Longformer-based encoder enables efficient modeling of long trajectories. PULSE-ICU was fine-tuned across 18 prediction tasks, including mortality, intervention forecasting, and phenotype identification, achieving strong performance across task types. External validation on eICU, HiRID, and P12 showed substantial improvements with minimal fine-tuning, demonstrating robustness to domain shift and variable constraints. These findings suggest that foundation-style modeling can improve data efficiency and adaptability, providing a scalable framework for ICU decision support across diverse clinical environments.

Figures (18)

• Figure 1: Illustration of irregular, heterogeneous ICU event sequences in EHR. Each row corresponds to a clinical variable (vital sign, laboratory test, procedure, medication), and each dot denotes a recorded event at its native timestamp.
• Figure 2: Representative ICU-focused foundation models for structured EHR data. For each study, we summarize the primary datasets, event embedding strategy, self-supervised pretraining objectives, backbone architecture, and major downstream clinical applications.
• Figure 3: Effect of fine-tuning data size on the downstream performance of PULSE-ICU in MIMIC-IV. Mean AUROC and AUPRC are plotted for binary classification, multi-class (SOFA; macro-averaged), multi-label (phenotype; macro-averaged), and the overall average across all tasks as the proportion of labeled fine-tuning data increases (0%, 10%, 30%, and 100%).
• Figure 4: Performance of the fine-tuned PULSE-ICU model under feature-limited conditions (MIMIC-Limited). Using only the 72 variables, AUROC (solid blue line) and AUPRC (dashed red line) are reported for binary, multi-class (SOFA), multi-label (phenotype), and overall task groups as the proportion of labeled fine-tuning data increases from 0% (zero-shot) to 100%.
• Figure 5: External validation and domain adaptation performance of the fine-tuned PULSE-ICU model across six target datasets: HiRID-first, HiRID-last, HiRID-4093, P12-24hr, P12-48hr, and eICU. For each dataset, AUROC (blue) and AUPRC (red) are plotted as functions of the proportion of target-domain data used for fine-tuning (0%–100%). Dashed horizontal lines denote the performance of models trained from random initialization using 100% of the target-domain data.