Comprehensive language-image pre-training for 3D medical image understanding

TL;DR

This work tackles data scarcity in 3D vision–language pre-training by introducing COLIPRI, a framework that combines CLIP-style global alignment with a report-generation objective and vision-only self-supervision to exploit both paired and unpaired 3D data. Through extensive ablations on report length, field of view, and patching, plus the integration of RRG and MAE objectives, COLIPRI achieves state-of-the-art performance in radiology report generation, classification probing, and zero-shot classification, while remaining competitive for segmentation. The approach demonstrates strong generalization across CT-RATE and RAD-ChestCT and provides a path toward holistic 3D medical understanding that supports radiologists with retrieval and first-pass interpretation. Limitations include prompt sensitivity in zero-shot and modest gains in segmentation, suggesting future work in more advanced MIM strategies and prompt-invariant representations to bolster dense-task performance.

Abstract

Vision-language pre-training, i.e., aligning images with paired text, is a powerful paradigm to create encoders that can be directly used for tasks such as classification and retrieval, and for downstream tasks such as segmentation and report generation. In the 3D medical image domain, these capabilities allow vision-language encoders (VLEs) to support radiologists by retrieving patients with similar abnormalities or predicting likelihoods of abnormality. While the methodology holds promise, data availability limits the capabilities of current 3D VLEs. In this paper, we alleviate the lack of data by injecting additional inductive biases: introducing a report generation objective and pairing vision-language pre-training with vision-only pre-training. This allows us to leverage both image-only and paired image-text 3D datasets, increasing the total amount of data to which our model is exposed. Through these additional inductive biases, paired with best practices of the 3D medical imaging domain, we develop the Comprehensive Language-image Pre-training (COLIPRI) encoder family. Our COLIPRI encoders achieve state-of-the-art performance in report generation, classification probing, and zero-shot classification, and remain competitive for semantic segmentation.

Figures (8)

• Figure 1: Out COLIPRI encoders yield general-purpose embeddings that can be adapted to a plethora of tasks, reaching state-of-the-art performance in multiple downstream tasks.
• Figure 2: COLIPRI: We investigate the combination of contrastive pre-training, report generation, and MIM to train 3D vision--language encoders.
• Figure 3: Reports of 3D images are substantially longer than those of 2D images, leading to a distribution shift between long reports seen during training and short prompts used for zero-shot classification. Additionally, long reports might allow the text encoder to overfit due to high dimensionality instead of learning semantics.
• Figure 4: Text generation pre-training. To yield more semantic image features, we feed them through cross-attention into our text generation Eva02 transformer architecture, tasked with Captioning or Parallel Captioning. This is optimised simultaneously with the CLIP objective.
• Figure 5: Report generation results. Our models enable generating reports of positive pathological findings substantially better than current models. In particular, the LLM trained on top of our models generate more accurate statements about abnormalities being present, as measured by RadFact-CT/$F_1$ (+). The exact values are presented in \ref{['tab:results_generation']}. RadFact-CT/$F_1$ (+/-): Positive and negative sentence accuracy is measured; RadFact-CT/$F_1$(+): only the positive sentences (i.e., describing an abnormality as judged by GPT) are measured.