Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding

TL;DR

This work addresses CT image interpretation by moving beyond global image-text alignment to fine-grained anatomy-level vision-language pre-training (fVLM). It introduces anatomy-wise decomposition of CT scans and radiology reports, with per-anatomy contrastive learning between visual and textual embeddings, and a dual false negative reduction strategy to mitigate mislabeling from normal and disease-like anatomies. Trained on the largest CT dataset to date, MedVL-CT69K, fVLM achieves state-of-the-art zero-shot abnormality detection across 54 diseases and 15 anatomies, and improves radiology report generation over CLIP baselines, demonstrated on MedVL-CT69K and public benchmarks CT-RATE and Rad-ChestCT. The approach offers improved interpretability and diagnostic versatility for clinical CT understanding, while future work aims at anatomy-wise report generation and reducing preprocessing overhead.

Abstract

Artificial intelligence (AI) shows great potential in assisting radiologists to improve the efficiency and accuracy of medical image interpretation and diagnosis. However, a versatile AI model requires large-scale data and comprehensive annotations, which are often impractical in medical settings. Recent studies leverage radiology reports as a naturally high-quality supervision for medical images, using contrastive language-image pre-training (CLIP) to develop language-informed models for radiological image interpretation. Nonetheless, these approaches typically contrast entire images with reports, neglecting the local associations between imaging regions and report sentences, which may undermine model performance and interoperability. In this paper, we propose a fine-grained vision-language model (fVLM) for anatomy-level CT image interpretation. Specifically, we explicitly match anatomical regions of CT images with corresponding descriptions in radiology reports and perform contrastive pre-training for each anatomy individually. Fine-grained alignment, however, faces considerable false-negative challenges, mainly from the abundance of anatomy-level healthy samples and similarly diseased abnormalities. To tackle this issue, we propose identifying false negatives of both normal and abnormal samples and calibrating contrastive learning from patient-level to disease-aware pairing. We curated the largest CT dataset to date, comprising imaging and report data from 69,086 patients, and conducted a comprehensive evaluation of 54 major and important disease diagnosis tasks across 15 main anatomies. Experimental results demonstrate the substantial potential of fVLM in versatile medical image interpretation. In the zero-shot classification task, we achieved an average AUC of 81.3% on 54 diagnosis tasks, surpassing CLIP and supervised methods by 12.9% and 8.0%, respectively.

Figures (13)

• Figure 1: Comparative analysis of vanilla VLM (CLIP) and our fine-grained VLM (fVLM). (a,b) A representative CT slice and its corresponding radiological report. (c,d) Visual activation maps generated by CLIP and fVLM respectively, illustrating regions of interest for pancreatitis diagnosis. (e) Quantitative comparison of AUC scores across 54 disease diagnosis tasks in 15 anatomies.
• Figure 2: Illustration of CT anatomy parsing (left) and diagnostic report decomposition (right).
• Figure 3: Framework of fVLM. (a) Visual encoding. We input a CT volume $I_i$ into the image encoder and extract corresponding visual tokens for each anatomy. We then append an anatomy-specific query token to the extracted visual tokens of each anatomy. These query tokens are subsequently updated through self-attention, constituting the visual representations of their respective anatomies. $N$ is the number of anatomies. (b) Textual encoding. We decompose the paired report $R_i$ into anatomy-wise descriptions and feed them separately into the text decoder to obtain anatomy-specific textual representation. (c) Fine-grained VLP. We perform local alignment for each individual anatomy across different CT scans. $L_j$ denotes the contrastive loss computed for the j-th anatomy.
• Figure 4: Illustration of the proposed dual false negative reduction approach. $V_{i,12}$ and $T_{i,12}$ represent pancreas-specific visual and textual features from the i-th sample, respectively. On one hand, we identify those samples not mentioned in the impression section of reports as normal and correct the corresponding labels for these semantically consistent samples to 1 in the label matrix. On the other hand, we further incorporate the estimated image-text similarities into the label matrix, aiming to capture potential semantic relationships between different samples and thereby enhance the model's semantic comprehension. Notably, to mitigate error accumulation in this process, we propose a co-teaching strategy, wherein two fVLMs are trained alternately and the image-text similarities employed by one model are estimated from the other.
• Figure 4: Effect of our proposed modules. CLIP serves as the baseline.