MGI: Multimodal Contrastive pre-training of Genomic and Medical Imaging

TL;DR

MGI tackles the challenge of multimodal cancer analysis by jointly learning from gene expression and medical imaging. It combines a ViT image encoder with a Mamba gene encoder and trains them with a self-supervised image-gene contrastive loss, followed by a lightweight fusion decoder for segmentation. The approach is evaluated on TCGA-LGG, where it achieves a Dice score of $0.901$, outperforming several unimodal baselines. This work demonstrates that aligning genomic and imaging representations can enhance tumor segmentation and suggests broader potential for genomics-informed medical imaging analysis.

Abstract

Medicine is inherently a multimodal discipline. Medical images can reflect the pathological changes of cancer and tumors, while the expression of specific genes can influence their morphological characteristics. However, most deep learning models employed for these medical tasks are unimodal, making predictions using either image data or genomic data exclusively. In this paper, we propose a multimodal pre-training framework that jointly incorporates genomics and medical images for downstream tasks. To address the issues of high computational complexity and difficulty in capturing long-range dependencies in genes sequence modeling with MLP or Transformer architectures, we utilize Mamba to model these long genomic sequences. We aligns medical images and genes using a self-supervised contrastive learning approach which combines the Mamba as a genetic encoder and the Vision Transformer (ViT) as a medical image encoder. We pre-trained on the TCGA dataset using paired gene expression data and imaging data, and fine-tuned it for downstream tumor segmentation tasks. The results show that our model outperformed a wide range of related methods.

Figures (2)

• Figure 1: MGI framework.
• Figure 2: Multimodal fusion module.