Arbitrary Data as Images: Fusion of Patient Data Across Modalities and Irregular Intervals with Vision Transformers

TL;DR

The approach, Vision Transformer for irregular sampled Multi-modal Measurements (ViTiMM), not only simplifies data preprocessing and modeling but also outperforms current state-of-the-art methods in predicting in-hospital mortality and phenotyping, as evaluated on 6,175 patients from the MIMIC-IV dataset.

Abstract

A patient undergoes multiple examinations in each hospital stay, where each provides different facets of the health status. These assessments include temporal data with varying sampling rates, discrete single-point measurements, therapeutic interventions such as medication administration, and images. While physicians are able to process and integrate diverse modalities intuitively, neural networks need specific modeling for each modality complicating the training procedure. We demonstrate that this complexity can be significantly reduced by visualizing all information as images along with unstructured text and subsequently training a conventional vision-text transformer. Our approach, Vision Transformer for irregular sampled Multi-modal Measurements (ViTiMM), not only simplifies data preprocessing and modeling but also outperforms current state-of-the-art methods in predicting in-hospital mortality and phenotyping, as evaluated on 6,175 patients from the MIMIC-IV dataset. The modalities include patient's clinical measurements, medications, X-ray images, and electrocardiography scans. We hope our work inspires advancements in multi-modal medical AI by reducing the training complexity to (visual) prompt engineering, thus lowering entry barriers and enabling no-code solutions for training. The source code will be made publicly available.

Figures (8)

• Figure 1: Example patient journey in the first 48 hours in the intensive care unit for hadm_id 28016813 from the MIMIC-IV dataset johnson2023mimiciv. The patient examinations include, among others, vital signs (e.g., blood pressure), laboratory measurements (e.g., glucose), medications (e.g., insulin), and additional tests such as X-ray images and electrocardiography scans. The measurements are recorded at irregular time intervals, which differ between patients, adding further complexity to the modeling process. Additionally, medications can be administered in various doses, which must also be considered.
• Figure 1: All clinical measurements (C) in their respective field color in the line plots.
• Figure 2: The four modalities from the MIMIC-IV dataset used in our model. We tested various formats (e.g., removing "*" markings, preprocessing ECGs), and found the visualized format performed best. a) Clinical measurements with x-axis representing 48 hours. b) Medications as cumulative dosage of the different drugs taken. c) 12-lead ECG with minimal preprocessing, from which 8 are shown. d) CXR image. The field names for clinical measurements can be found in Supplementary Figure \ref{['fig:lab-parameters-image']} and for medications in Supplementary Figure \ref{['fig:med-parameters-image']}. The 12 leads of the ECG are in standard chronological order gow2023mimicivecg.
• Figure 2: All medications (M) in their respective field color in the line plots.
• Figure 3: Interpretability with Visualizing Attention Maps for in-hospital mortality prediction for hadm_id = 30315583. The patient has died within the hospital stay, which the model rightfully predicted. The model clearly focuses on specific attributes in the clinical measurements and medications. In contrast, the attention for ECG appears more diffuse, suggesting that the model leverages a broader, more global context. For the CXR image, the attention is particularly concentrated on the implanted device. More examples can be found in the Supplemental Material.