Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Predicting Stroke through Retinal Graphs and Multimodal Self-supervised Learning

Yuqing Huang, Bastian Wittmann, Olga Demler, Bjoern Menze, Neda Davoudi

TL;DR

Findings indicate that retinal images are a cost-effective method for improving cardiovascular disease predictions and pave the way for future research into retinal and cerebral vessel connections and the use of graph-based retinal vessel representations.

Abstract

Early identification of stroke is crucial for intervention, requiring reliable models. We proposed an efficient retinal image representation together with clinical information to capture a comprehensive overview of cardiovascular health, leveraging large multimodal datasets for new medical insights. Our approach is one of the first contrastive frameworks that integrates graph and tabular data, using vessel graphs derived from retinal images for efficient representation. This method, combined with multimodal contrastive learning, significantly enhances stroke prediction accuracy by integrating data from multiple sources and using contrastive learning for transfer learning. The self-supervised learning techniques employed allow the model to learn effectively from unlabeled data, reducing the dependency on large annotated datasets. Our framework showed an AUROC improvement of 3.78% from supervised to self-supervised approaches. Additionally, the graph-level representation approach achieved superior performance to image encoders while significantly reducing pre-training and fine-tuning runtimes. These findings indicate that retinal images are a cost-effective method for improving cardiovascular disease predictions and pave the way for future research into retinal and cerebral vessel connections and the use of graph-based retinal vessel representations.

Predicting Stroke through Retinal Graphs and Multimodal Self-supervised Learning

TL;DR

Findings indicate that retinal images are a cost-effective method for improving cardiovascular disease predictions and pave the way for future research into retinal and cerebral vessel connections and the use of graph-based retinal vessel representations.

Abstract

Early identification of stroke is crucial for intervention, requiring reliable models. We proposed an efficient retinal image representation together with clinical information to capture a comprehensive overview of cardiovascular health, leveraging large multimodal datasets for new medical insights. Our approach is one of the first contrastive frameworks that integrates graph and tabular data, using vessel graphs derived from retinal images for efficient representation. This method, combined with multimodal contrastive learning, significantly enhances stroke prediction accuracy by integrating data from multiple sources and using contrastive learning for transfer learning. The self-supervised learning techniques employed allow the model to learn effectively from unlabeled data, reducing the dependency on large annotated datasets. Our framework showed an AUROC improvement of 3.78% from supervised to self-supervised approaches. Additionally, the graph-level representation approach achieved superior performance to image encoders while significantly reducing pre-training and fine-tuning runtimes. These findings indicate that retinal images are a cost-effective method for improving cardiovascular disease predictions and pave the way for future research into retinal and cerebral vessel connections and the use of graph-based retinal vessel representations.

Paper Structure

This paper contains 19 sections, 3 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Multimodal learning
  3. Self-supervised Learning
  4. Graph Representation of Vessels
  5. Contrastive Learning Frameworks
  6. Raw Fundus Photographs
  7. Retinal Vessel Probability Mask
  8. Retinal Vessel Graph
  9. Experiments and Results
  10. Dataset
  11. Experimental Setup
  12. Tabular-NN
  13. Imaging-NN
  14. Multimodal-NN
  15. Stroke Prediction
  16. ...and 4 more sections

Figures (5)

  • Figure 1: The overall contrastive learning framework with fundus photographs and tabular data. The Module in the dotted box will be further specified in the subsections depending on the strategy used.
  • Figure 2: Module that directly learns from raw fundus images.
  • Figure 3: Module that extracts probabilistic masks from fundus images.
  • Figure 4: Module that turns fundus images into vessel segmentations and further into graph representations. A graph encoder is used for feature extraction from graphs.
  • Figure 5: Comparison of test-time ROC curves for Imaging-NN, Multimodal-NN, and Multimodal-CL-graph methods.