Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Specialist to Generalist: Unlocking SAM's Learning Potential on Unlabeled Medical Images

Vi Vu, Thanh-Huy Nguyen, Tien-Thinh Nguyen, Ba-Thinh Lam, Hoang-Thien Nguyen, Tianyang Wang, Xingjian Li, Min Xu

TL;DR

The paper tackles the challenge of adapting foundation models like SAM to medical imaging under domain shift and limited labels by introducing SC-SAM, a specialist-generalist framework where a U-Net acts as a specialist to guide SAM with point prompts and pseudo-labels, and SAM serves as a semantic regularizer for the U-Net. A bidirectional co-training loop enables both models to learn From unlabeled data, stabilized by a sigmoid ramp-up that mitigates early overfitting. Across PROMISE12 and COLON benchmarks, SC-SAM achieves state-of-the-art results, outperforming existing semi-supervised SAM variants and even strong medical foundation models, demonstrating the value of combining traditional specialists with modern generalist models for label-efficient medical segmentation. The work highlights a practical pathway to leverage unlabeled medical data through specialist-generalist cooperation, with code available for reproduction.

Abstract

Foundation models like the Segment Anything Model (SAM) show strong generalization, yet adapting them to medical images remains difficult due to domain shift, scarce labels, and the inability of Parameter-Efficient Fine-Tuning (PEFT) to exploit unlabeled data. While conventional models like U-Net excel in semi-supervised medical learning, their potential to assist a PEFT SAM has been largely overlooked. We introduce SC-SAM, a specialist-generalist framework where U-Net provides point-based prompts and pseudo-labels to guide SAM's adaptation, while SAM serves as a powerful generalist supervisor to regularize U-Net. This reciprocal guidance forms a bidirectional co-training loop that allows both models to effectively exploit the unlabeled data. Across prostate MRI and polyp segmentation benchmarks, our method achieves state-of-the-art results, outperforming other existing semi-supervised SAM variants and even medical foundation models like MedSAM, highlighting the value of specialist-generalist cooperation for label-efficient medical image segmentation. Our code is available at https://github.com/vnlvi2k3/SC-SAM.

From Specialist to Generalist: Unlocking SAM's Learning Potential on Unlabeled Medical Images

TL;DR

The paper tackles the challenge of adapting foundation models like SAM to medical imaging under domain shift and limited labels by introducing SC-SAM, a specialist-generalist framework where a U-Net acts as a specialist to guide SAM with point prompts and pseudo-labels, and SAM serves as a semantic regularizer for the U-Net. A bidirectional co-training loop enables both models to learn From unlabeled data, stabilized by a sigmoid ramp-up that mitigates early overfitting. Across PROMISE12 and COLON benchmarks, SC-SAM achieves state-of-the-art results, outperforming existing semi-supervised SAM variants and even strong medical foundation models, demonstrating the value of combining traditional specialists with modern generalist models for label-efficient medical segmentation. The work highlights a practical pathway to leverage unlabeled medical data through specialist-generalist cooperation, with code available for reproduction.

Abstract

Foundation models like the Segment Anything Model (SAM) show strong generalization, yet adapting them to medical images remains difficult due to domain shift, scarce labels, and the inability of Parameter-Efficient Fine-Tuning (PEFT) to exploit unlabeled data. While conventional models like U-Net excel in semi-supervised medical learning, their potential to assist a PEFT SAM has been largely overlooked. We introduce SC-SAM, a specialist-generalist framework where U-Net provides point-based prompts and pseudo-labels to guide SAM's adaptation, while SAM serves as a powerful generalist supervisor to regularize U-Net. This reciprocal guidance forms a bidirectional co-training loop that allows both models to effectively exploit the unlabeled data. Across prostate MRI and polyp segmentation benchmarks, our method achieves state-of-the-art results, outperforming other existing semi-supervised SAM variants and even medical foundation models like MedSAM, highlighting the value of specialist-generalist cooperation for label-efficient medical image segmentation. Our code is available at https://github.com/vnlvi2k3/SC-SAM.
Paper Structure (9 sections, 20 equations, 1 figure, 4 tables)

This paper contains 9 sections, 20 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Experiment results
  4. Datasets
  5. Quantitative results
  6. Qualitative results
  7. Ablation Study
  8. Conclusion
  9. Acknowledgment

Figures (1)

  • Figure 1: Overview of different techniques for incorporating SAM in semi-supervised settings: a) PEFT-SAM, b) Dual-SAM, c) SP-SAM, and d) SC-SAM (Ours)