SeLIP: Similarity Enhanced Contrastive Language Image Pretraining for Multi-modal Head MRI

TL;DR

SeLIP tackles data scarcity in medical imaging by tying head MRI to radiology reports through a contrastive pretraining framework that incorporates a mixed syntax and semantic similarity target. It leverages structured information extraction via LLMs to create modality-specific JSON descriptions and blends softened semantic targets with the CLIP loss to handle medical text similarities. On the TFCH head MRI dataset, SeLIP improves image-text retrieval, classification (including cross-domain BraTS19), and segmentation, demonstrating strong cross-domain generalization and the practical viability of text-informed medical foundation models. By exploiting textual similarities to guide multimodal representation learning, SeLIP advances medical vision-language pretraining and enables more effective models with limited data.

Abstract

Despite that deep learning (DL) methods have presented tremendous potential in many medical image analysis tasks, the practical applications of medical DL models are limited due to the lack of enough data samples with manual annotations. By noting that the clinical radiology examinations are associated with radiology reports that describe the images, we propose to develop a foundation model for multi-model head MRI by using contrastive learning on the images and the corresponding radiology findings. In particular, a contrastive learning framework is proposed, where a mixed syntax and semantic similarity matching metric is integrated to reduce the thirst of extreme large dataset in conventional contrastive learning framework. Our proposed similarity enhanced contrastive language image pretraining (SeLIP) is able to effectively extract more useful features. Experiments revealed that our proposed SeLIP performs well in many downstream tasks including image-text retrieval task, classification task, and image segmentation, which highlights the importance of considering the similarities among texts describing different images in developing medical image foundation models.

Figures (6)

• Figure 1: Examples of the "Findings" descriptions and the extracted structural information.
• Figure 2: Procedure of structural information extraction using LLM, which includes two phases, i.e., pseudo-data generation and finetune. Pseudo reports are generated from random JSON items, and then used to train the LLM to generate the JSON items from the corresponding pseudo reports.
• Figure 3: Structure of our proposed SeLIP for multi-modal MRI and radiology findings alignment pretraining.
• Figure 4: Examples of radiology finding statements pairs that may confuse the network learning.
• Figure 5: Examples of the cosine similarity measurements between an image and candidate text descriptions by using ResNet50 as backbone. The real lesions on the image are identified by yellow arrows.