Enhancing SAM with Efficient Prompting and Preference Optimization for Semi-supervised Medical Image Segmentation

TL;DR

This work tackles the data-hungry nature of medical image segmentation by extending SAM with annotation-efficient, unsupervised prompts derived from BiomedCLIP, MedVInT, and GPT-4, capturing semantic, locational, and structural cues. It then introduces a Direct Preference Optimization–based virtual annotator to simulate human feedback and guide end-to-end training on largely unlabeled data, removing the need for an explicit reward function. Across lung, breast tumor, and abdominal organ segmentation tasks in X-ray, ultrasound, and CT, the approach achieves state-of-the-art performance in low-annotation regimes, demonstrating strong generalization and robustness to rating noise. The method reduces annotation costs while maintaining high fidelity segmentations, with practical impact for scalable clinical imaging analysis.

Abstract

Foundational models such as the Segment Anything Model (SAM) are gaining traction in medical imaging segmentation, supporting multiple downstream tasks. However, such models are supervised in nature, still relying on large annotated datasets or prompts supplied by experts. Conventional techniques such as active learning to alleviate such limitations are limited in scope and still necessitate continuous human involvement and complex domain knowledge for label refinement or establishing reward ground truth. To address these challenges, we propose an enhanced Segment Anything Model (SAM) framework that utilizes annotation-efficient prompts generated in a fully unsupervised fashion, while still capturing essential semantic, location, and shape information through contrastive language-image pretraining and visual question answering. We adopt the direct preference optimization technique to design an optimal policy that enables the model to generate high-fidelity segmentations with simple ratings or rankings provided by a virtual annotator simulating the human annotation process. State-of-the-art performance of our framework in tasks such as lung segmentation, breast tumor segmentation, and organ segmentation across various modalities, including X-ray, ultrasound, and abdominal CT, justifies its effectiveness in low-annotation data scenarios.

Figures (7)

• Figure 1: Overview of our model:(a) SAM and SAM-based approaches rely on expert prompts during both training and inference. (b) SAM-SP samsp introduces a self-promoting module, eliminating the need for expert prompts during inference. (c) MedCLIP-SAM medclipsam uses unsupervised semantic prompts to generate pseudo-labels through SAM. (d) Our approach not only combines semantic, location, and generic information via unsupervised prompts but also introduces a preference-based alignment module to reward or penalize the model.
• Figure 2: Illustration of the proposed framework for semi-supervised segmentation: Unsupervised geometric and text prompts, obtained from pretrained BiomedCLIP, MedVInT, and GPT-4 models, are fed into the prompt encoder for finetuning the framework on a small fraction of annotated data. In the next stage, we simulate a virtual annotation process that assigns ratings to the generated segmentation candidates, which are used to fine-tune the decoder. This stage handles unannotated data, as the model does not rely on ground truth for direct supervision but only for rating while simulating a human annotator's feedback.
• Figure 3: Quantitative comparison with SOTA. Dice score (for Chest Xray, Breast USD) and mean Dice score (for AMOS CT) have been shown to measure the model segmentation performance on different proportions of training data (10%, 20%, 50%, and 100%).
• Figure 4: Qualitative comparisons were made between the segmentation results of nnUnet, SAM-Med2D, and our framework on 2D datasets. BiomedCLIP-based saliency maps are also depicted. Experiments were conducted in 50% data settings.
• Figure 5: Segmentation maps of different anatomical structures (liver and spleen) for SAM-Med3D and our method