Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Modal Contrastive Learning for Online Clinical Time-Series Applications

Fabian Baldenweg, Manuel Burger, Gunnar Rätsch, Rita Kuznetsova

TL;DR

This work applies advanced self-supervised multi-modal contrastive learning techniques to ICU data, specifically focusing on clinical notes and time-series for clinically relevant online prediction tasks.

Abstract

Electronic Health Record (EHR) datasets from Intensive Care Units (ICU) contain a diverse set of data modalities. While prior works have successfully leveraged multiple modalities in supervised settings, we apply advanced self-supervised multi-modal contrastive learning techniques to ICU data, specifically focusing on clinical notes and time-series for clinically relevant online prediction tasks. We introduce a loss function Multi-Modal Neighborhood Contrastive Loss (MM-NCL), a soft neighborhood function, and showcase the excellent linear probe and zero-shot performance of our approach.

Multi-Modal Contrastive Learning for Online Clinical Time-Series Applications

TL;DR

This work applies advanced self-supervised multi-modal contrastive learning techniques to ICU data, specifically focusing on clinical notes and time-series for clinically relevant online prediction tasks.

Abstract

Electronic Health Record (EHR) datasets from Intensive Care Units (ICU) contain a diverse set of data modalities. While prior works have successfully leveraged multiple modalities in supervised settings, we apply advanced self-supervised multi-modal contrastive learning techniques to ICU data, specifically focusing on clinical notes and time-series for clinically relevant online prediction tasks. We introduce a loss function Multi-Modal Neighborhood Contrastive Loss (MM-NCL), a soft neighborhood function, and showcase the excellent linear probe and zero-shot performance of our approach.
Paper Structure (23 sections, 4 equations, 3 figures, 4 tables)

This paper contains 23 sections, 4 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Learning on Medical Time-Series
  4. Multi-modal Learning
  5. Methods
  6. Notation
  7. Model Architecture
  8. Loss
  9. Results and Discussion
  10. Experimental Setup
  11. Evaluation
  12. Results
  13. Scarce label setting
  14. Ablation on Note Types
  15. Conclusion
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Training pipeline
  • Figure 2: AuPRC when training with reduced labels, x-axis shows the percentage of of labels used from the full training set. All results were obtained using the same time-series architecture. We mark the percentage, where supervised outperforms zero-shot.
  • Figure 3: Zero-shot AuPRC for mortality (blue) and decompensation (orange) for different sets of note types for MM-NCL. We greedily remove note types from the left to the right based on mortality (Fig. \ref{['subfig:note_abl_mort']}) and decompensation (Fig. \ref{['subfig:note_abl_decomp']}) AuPRC. Removing the last remaining category (Nursing/other in both cases) leaves no training data for the text modality, so there is no result in the rightmost columns.