Global Contrastive Training for Multimodal Electronic Health Records with Language Supervision

TL;DR

This work tackles the challenge of learning from multimodal electronic health records by introducing a global contrastive framework that aligns a patient’s multimodal representation (medical time series and clinical notes) with discharge summaries. It combines temporal cross-attention transformers with a dynamic embedding and tokenization scheme (including flexible positional encoding, Time2Vec time encoding, and variable-specific encoders) and a cross-modal fusion module to form $h_{time+note}$ representations. The objective integrates a global contrastive loss, with $\ l L_{alignment} = \mathcal{L}_{MD} + \mathcal{L}_{DM}$, and a total loss $\mathcal{L}_{total} = \alpha \mathcal{L}_{alignment} + \beta \mathcal{L}_{ce}$, enabling learning that couples multimodal features to discharge summaries; improvements are further pursued by generating LLM-derived descriptions of time-series dynamics. Experiments on a real-world UF Health dataset of $113{,}953$ adults and $124{,}777$ surgeries show state-of-the-art mean AUROC across nine postoperative complications, with LLM-enhanced discharge summaries yielding the best performance ($\approx 0.862$). The approach scales to additional modalities and provides a practical, text-supervised pathway to exploit holistic patient narratives for better predictive accuracy in clinical settings.

Abstract

Modern electronic health records (EHRs) hold immense promise in tracking personalized patient health trajectories through sequential deep learning, owing to their extensive breadth, scale, and temporal granularity. Nonetheless, how to effectively leverage multiple modalities from EHRs poses significant challenges, given its complex characteristics such as high dimensionality, multimodality, sparsity, varied recording frequencies, and temporal irregularities. To this end, this paper introduces a novel multimodal contrastive learning framework, specifically focusing on medical time series and clinical notes. To tackle the challenge of sparsity and irregular time intervals in medical time series, the framework integrates temporal cross-attention transformers with a dynamic embedding and tokenization scheme for learning multimodal feature representations. To harness the interconnected relationships between medical time series and clinical notes, the framework equips a global contrastive loss, aligning a patient's multimodal feature representations with the corresponding discharge summaries. Since discharge summaries uniquely pertain to individual patients and represent a holistic view of the patient's hospital stay, machine learning models are led to learn discriminative multimodal features via global contrasting. Extensive experiments with a real-world EHR dataset demonstrated that our framework outperformed state-of-the-art approaches on the exemplar task of predicting the occurrence of nine postoperative complications for more than 120,000 major inpatient surgeries using multimodal data from UF health system split among three hospitals (UF Health Gainesville, UF Health Jacksonville, and UF Health Jacksonville-North).

Figures (3)

• Figure 1: An example of unaligned, though clinically relevant, clinical notes and medical time series, where clinical notes are taken beginning patient admission through final discharge, including diverse and comprehensive information while medical time series only reflecting the patient's vital signs during a major surgery. This is ubiquitous in real-world EHR datasets.
• Figure 2: The overview of the proposed global contrastive learning framework for multimodal EHRs. The framework consists of three main components: (1) Feature learning sub-network extracts temporal representation from medical time series and textual representation from clinical notes. (2) cross-modal fusion sub-network merges unimodal features with transformers in the cross-attention fashion, in which feature embedding from one modality is enriched by searching for the most relevant feature in the other modality; (3) modal optimization uses the learning objective combining both the cross-entropy loss between predictions and ground truth labels, and the contrastive loss aligning multimodal representations with discharge summaries improved by LLMs.
• Figure 3: Prompting LLMs generating in-context texts for medical time series.