MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation

TL;DR

This work presents MedCLIPSeg, a novel framework that adapts CLIP for robust, data-efficient, and uncertainty-aware medical image segmentation, and demonstrates the potential of probabilistic vision-language modeling for text-driven medical image segmentation.

Abstract

Medical image segmentation remains challenging due to limited annotations for training, ambiguous anatomical features, and domain shifts. While vision-language models such as CLIP offer strong cross-modal representations, their potential for dense, text-guided medical image segmentation remains underexplored. We present MedCLIPSeg, a novel framework that adapts CLIP for robust, data-efficient, and uncertainty-aware medical image segmentation. Our approach leverages patch-level CLIP embeddings through probabilistic cross-modal attention, enabling bidirectional interaction between image and text tokens and explicit modeling of predictive uncertainty. Together with a soft patch-level contrastive loss that encourages more nuanced semantic learning across diverse textual prompts, MedCLIPSeg effectively improves data efficiency and domain generalizability. Extensive experiments across 16 datasets spanning five imaging modalities and six organs demonstrate that MedCLIPSeg outperforms prior methods in accuracy, efficiency, and robustness, while providing interpretable uncertainty maps that highlight local reliability of segmentation results. This work demonstrates the potential of probabilistic vision-language modeling for text-driven medical image segmentation.

Figures (10)

• Figure 1: (Top): Comparison between deterministic and probabilistic cross-modal fusion techniques in CLIP adaptation for text-driven segmentation. Probabilistic formulation models variability in visual–textual representations as distributions, enabling more robust feature alignment. (Bottom): Robustness and Reliability plots over ID and OOD data show improved generalization, with smaller out-of-domain performance drops and better calibration of predicted confidence, reflected by lower Brier scores.
• Figure 2: Overview of the proposed MedCLIPSeg framework for text-driven medical image segmentation. The model extends CLIP with vision and language encoders connected via PVL Adapters, which perform confidence-weighted image–text fusion at multiple deep layers. Segmentation and uncertainty maps arise from the mean and entropy of posterior samples, with a soft patch-level contrastive loss.
• Figure 3: Illustrations of PVL Adapter and $\texttt{Attn}_{\texttt{PVL}}$.
• Figure 4: Segmentation and uncertainty visualizations. Uncertainty peaks along lesion boundaries and remains consistent across diverse datasets, indicating reliable calibration and generalization. ID data are in blue while OOD data are in red.
• Figure 5: Layer-wise interventions (left) and confidence weighting ($\beta$) (right) ablations averaged on ID and OOD data.