Towards Universal Text-driven CT Image Segmentation

TL;DR

OpenVocabCT introduces a vision–language CT segmentation framework trained on a large-scale CT–report corpus (CT-RATE) with organ-level captions generated by LLMs. It uses multi-granularity contrastive learning to align image and text across multiple caption granularities, and pretrains the image encoder on segmentation data before text-guided fine-tuning. The method achieves superior organ and tumor segmentation across multiple public datasets and demonstrates robust generalization to unseen and synonym-based prompts, outperforming both vision-only and prior text-driven baselines. This approach enables effective open-vocabulary, text-prompt CT segmentation with strong potential for clinical deployment and prompt-based interaction in real-world workflows.

Abstract

Computed tomography (CT) is extensively used for accurate visualization and segmentation of organs and lesions. While deep learning models such as convolutional neural networks (CNNs) and vision transformers (ViTs) have significantly improved CT image analysis, their performance often declines when applied to diverse, real-world clinical data. Although foundation models offer a broader and more adaptable solution, their potential is limited due to the challenge of obtaining large-scale, voxel-level annotations for medical images. In response to these challenges, prompting-based models using visual or text prompts have emerged. Visual-prompting methods, such as the Segment Anything Model (SAM), still require significant manual input and can introduce ambiguity when applied to clinical scenarios. Instead, foundation models that use text prompts offer a more versatile and clinically relevant approach. Notably, current text-prompt models, such as the CLIP-Driven Universal Model, are limited to text prompts already encountered during training and struggle to process the complex and diverse scenarios of real-world clinical applications. Instead of fine-tuning models trained from natural imaging, we propose OpenVocabCT, a vision-language model pretrained on large-scale 3D CT images for universal text-driven segmentation. Using the large-scale CT-RATE dataset, we decompose the diagnostic reports into fine-grained, organ-level descriptions using large language models for multi-granular contrastive learning. We evaluate our OpenVocabCT on downstream segmentation tasks across nine public datasets for organ and tumor segmentation, demonstrating the superior performance of our model compared to existing methods. All code, datasets, and models will be publicly released at https://github.com/ricklisz/OpenVocabCT.

Figures (6)

• Figure 1: We use CT-RATE hamamci2024developing, a large-scale paired CT and radiology report dataset, to generate pre-training data. Some key information about this dataset is present: (a) Example of paired CT scans and detailed captions for each organ; (b) Top 100 captions containing detailed organ-level information; (c) Distribution of the top 20 organs/tissues from these captions. We leverage pre-trained large language models to break down radiology reports into fine-grained captions and filtered out low-quality captions using substring matching for open-vocabulary segmentation training.
• Figure 2: Overall workflow for OpenVocabCT. (a) We first curate granular organ-level captions from CT-RATE’s radiology report using LLMs with few-shot examples. The LLMs break down long radiology findings into organ-level captions, which are further filtered via substring matching to our metadata. (b) We pretrain our vision language model using a multi-granularity contrastive loss. Each CT image is paired with multiple granular captions and the original report to enhance text representation learning. (c) We finetune the vision language model on CT segmentation datasets with text prompts for each organ.
• Figure 3: Visualization of organ segmentation on TotalSegmentator. Each segmentation model is evaluated under training seen prompts.
• Figure 4: Visualization of tumor segmentation on MSD Pancreas, MSD Colon and KiTS. Each segmentation model is evaluated under training seen prompts.
• Figure 5: Visualization of generalization study (axial view). Each segmentation model is evaluated under training unseen prompts, as depicted in the corresponding color-coded legend.