Medical SAM3: A Foundation Model for Universal Prompt-Driven Medical Image Segmentation

TL;DR

Medical SAM3 tackles the domain-shift challenge in medical image segmentation by fully fine-tuning a SAM3-based foundation model on a large, diverse corpus of 2D and 3D medical data with text prompts and masks. By unifying inputs to a single 2D representation, applying stratified tuning, and enforcing text-driven semantic alignment alongside a set-prediction objective, the approach removes dependence on privileged spatial prompts and achieves strong generalization across organs, modalities, and dimensionalities. It delivers substantial improvements over vanilla SAM3 on internal held-out tasks and, crucially, demonstrates robust zero-shot performance on external datasets spanning ultrasound, endoscopy, and fundus imaging, including challenging small and low-contrast structures. The work highlights the importance of holistic representation learning for reliable prompt-driven medical segmentation and provides a large, open corpus and codebase to enable broader deployment and evaluation.

Abstract

Promptable segmentation foundation models such as SAM3 have demonstrated strong generalization capabilities through interactive and concept-based prompting. However, their direct applicability to medical image segmentation remains limited by severe domain shifts, the absence of privileged spatial prompts, and the need to reason over complex anatomical and volumetric structures. Here we present Medical SAM3, a foundation model for universal prompt-driven medical image segmentation, obtained by fully fine-tuning SAM3 on large-scale, heterogeneous 2D and 3D medical imaging datasets with paired segmentation masks and text prompts. Through a systematic analysis of vanilla SAM3, we observe that its performance degrades substantially on medical data, with its apparent competitiveness largely relying on strong geometric priors such as ground-truth-derived bounding boxes. These findings motivate full model adaptation beyond prompt engineering alone. By fine-tuning SAM3's model parameters on 33 datasets spanning 10 medical imaging modalities, Medical SAM3 acquires robust domain-specific representations while preserving prompt-driven flexibility. Extensive experiments across organs, imaging modalities, and dimensionalities demonstrate consistent and significant performance gains, particularly in challenging scenarios characterized by semantic ambiguity, complex morphology, and long-range 3D context. Our results establish Medical SAM3 as a universal, text-guided segmentation foundation model for medical imaging and highlight the importance of holistic model adaptation for achieving robust prompt-driven segmentation under severe domain shift. Code and model will be made available at https://github.com/AIM-Research-Lab/Medical-SAM3.

Figures (4)

• Figure 1: Universal medical image segmentation via text prompting with Medical SAM3. Our proposed model unifies diverse medical imaging modalities—ranging from radiology (CT, MRI, X-Ray) to optical imaging (Fundus, Dermoscopy, Endoscopy) and pathology—into a single framework.
• Figure 2: Overview of Medical SAM3. Medical SAM3 takes a text prompt and medical images (2D or slice-based 3D) as input. A detector segments target instances in the current frame, while an optional tracker propagates masks across frames via a memory bank. The final prediction is produced by merging detected and propagated masks, supporting semantic-driven segmentation without privileged spatial prompts.
• Figure 3: Radar chart overview of segmentation performance. Results are split by internal validation (top) and external generalization (bottom), reporting Dice (left) and IoU (right) scores. The red area (Medical SAM3) significantly covers the blue area (SAM3) in all scenarios, aligning with the metrics in Table \ref{['tab:inter_exter_valid']}.
• Figure 4: Visualization of the segmentation performance of SAM3 and Medical SAM3