Cross Prompting Consistency with Segment Anything Model for Semi-supervised Medical Image Segmentation

TL;DR

The paper tackles semi-supervised medical image segmentation under scarce labels by leveraging the prompting flexibility of the Segment Anything Model (SAM). It introduces CPC-SAM, a cross prompting framework with a dual-branch SAM that generates prompts from unprompted branch outputs and uses cross‑branch supervision, augmented by a prompt consistency regularization (PCR) to stabilize outputs across varying prompts; the learning objective is $L_{total} = L_s + \lambda_{1} L_{cross}^u + \lambda_{2} L_c^u$, where $L_{cross}^u$ is defined as the symmetric Dice/CE cross-prompt loss and $L_c^u$ enforcing cross-prompt invariance. Empirical results on BUSI and ACDC show substantial improvements, including a >9% Dice gain on BUSI with only 10 labeled ultrasound images and top performance across metrics on ACDC. The work demonstrates effective integration of a foundation model into SSL for medical imaging and suggests directions for richer prompts and more robust prompting strategies.

Abstract

Semi-supervised learning (SSL) has achieved notable progress in medical image segmentation. To achieve effective SSL, a model needs to be able to efficiently learn from limited labeled data and effectively exploiting knowledge from abundant unlabeled data. Recent developments in visual foundation models, such as the Segment Anything Model (SAM), have demonstrated remarkable adaptability with improved sample efficiency. To harness the power of foundation models for application in SSL, we propose a cross prompting consistency method with segment anything model (CPC-SAM) for semi-supervised medical image segmentation. Our method employs SAM's unique prompt design and innovates a cross-prompting strategy within a dual-branch framework to automatically generate prompts and supervisions across two decoder branches, enabling effectively learning from both scarce labeled and valuable unlabeled data. We further design a novel prompt consistency regularization, to reduce the prompt position sensitivity and to enhance the output invariance under different prompts. We validate our method on two medical image segmentation tasks. The extensive experiments with different labeled-data ratios and modalities demonstrate the superiority of our proposed method over the state-of-the-art SSL methods, with more than 9% Dice improvement on the breast cancer segmentation task.

Figures (5)

• Figure 1: The overview of our proposed method. The adapted dual-branch SAM is fine-tuned by the cross prompting loss $L_{cross}^u$ with a prompt consistency regularization $L_c^u$ on the unlabeled data in addition to the supervised loss.
• Figure 2: Visualizations of different methods on the BUSI dataset with 10 labeled images (top) and on the ACDC dataset with 1 labeled patient (bottom).
• Figure 2: Ablation studies of different components of our method on the ACDC dataset.
• Figure 3: Effects of different numbers of center and random points in the PCR on the ACDC dataset.
• Figure 4: Effects of different architectures.