Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MedSAM-U: Uncertainty-Guided Auto Multi-Prompt Adaptation for Reliable MedSAM

Nan Zhou, Ke Zou, Kai Ren, Mengting Luo, Linchao He, Meng Wang, Yidi Chen, Yi Zhang, Hu Chen, Huazhu Fu

TL;DR

This paper introduces MedSAM-U, an uncertainty-guided framework designed to automatically refine multi-prompt inputs for more reliable and precise medical image segmentation, and employs uncertainty-guided multi-prompt to effectively estimate the uncertainties associated with the prompts and their initial segmentation results.

Abstract

The Medical Segment Anything Model (MedSAM) has shown remarkable performance in medical image segmentation, drawing significant attention in the field. However, its sensitivity to varying prompt types and locations poses challenges. This paper addresses these challenges by focusing on the development of reliable prompts that enhance MedSAM's accuracy. We introduce MedSAM-U, an uncertainty-guided framework designed to automatically refine multi-prompt inputs for more reliable and precise medical image segmentation. Specifically, we first train a Multi-Prompt Adapter integrated with MedSAM, creating MPA-MedSAM, to adapt to diverse multi-prompt inputs. We then employ uncertainty-guided multi-prompt to effectively estimate the uncertainties associated with the prompts and their initial segmentation results. In particular, a novel uncertainty-guided prompts adaptation technique is then applied automatically to derive reliable prompts and their corresponding segmentation outcomes. We validate MedSAM-U using datasets from multiple modalities to train a universal image segmentation model. Compared to MedSAM, experimental results on five distinct modal datasets demonstrate that the proposed MedSAM-U achieves an average performance improvement of 1.7\% to 20.5\% across uncertainty-guided prompts.

MedSAM-U: Uncertainty-Guided Auto Multi-Prompt Adaptation for Reliable MedSAM

TL;DR

This paper introduces MedSAM-U, an uncertainty-guided framework designed to automatically refine multi-prompt inputs for more reliable and precise medical image segmentation, and employs uncertainty-guided multi-prompt to effectively estimate the uncertainties associated with the prompts and their initial segmentation results.

Abstract

The Medical Segment Anything Model (MedSAM) has shown remarkable performance in medical image segmentation, drawing significant attention in the field. However, its sensitivity to varying prompt types and locations poses challenges. This paper addresses these challenges by focusing on the development of reliable prompts that enhance MedSAM's accuracy. We introduce MedSAM-U, an uncertainty-guided framework designed to automatically refine multi-prompt inputs for more reliable and precise medical image segmentation. Specifically, we first train a Multi-Prompt Adapter integrated with MedSAM, creating MPA-MedSAM, to adapt to diverse multi-prompt inputs. We then employ uncertainty-guided multi-prompt to effectively estimate the uncertainties associated with the prompts and their initial segmentation results. In particular, a novel uncertainty-guided prompts adaptation technique is then applied automatically to derive reliable prompts and their corresponding segmentation outcomes. We validate MedSAM-U using datasets from multiple modalities to train a universal image segmentation model. Compared to MedSAM, experimental results on five distinct modal datasets demonstrate that the proposed MedSAM-U achieves an average performance improvement of 1.7\% to 20.5\% across uncertainty-guided prompts.
Paper Structure (19 sections, 20 equations, 4 figures, 4 tables, 1 algorithm)

This paper contains 19 sections, 20 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related works
  3. SAM for medical image segmentation
  4. Prompts-based methods for medical image segmentation
  5. Uncertainty-based methods for medical image segmentation
  6. Method
  7. Multi-Prompt Adaptation for MedSAM
  8. Uncertainty estimation of MPA-MedSAM with multi-type prompts
  9. Uncertainty-Guided Prompts Adaptation
  10. Experiments and Results
  11. Datasets & Loss & Implementation Details
  12. Datasets
  13. Loss function
  14. Implementation Details
  15. Comparisons with the SOTA methods on Multi-modality Images
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Comparison of Dice Results Under (a) Bboxes with different overlap ratio on SAM and MedSAM Models. (b) 1) Previous methods predict annotation mask in only single step; 2) MedSAM-U, our method, automatically refine prompts with uncertainty-guided multi-prompt adaptation for reliable MedSAM. Example is from Colonoscopy test set pogorelov2017kvasir.
  • Figure 2: The overall framework of MedSAM-U. The framework is presented through three key illustrations: (a) the training process of MPA-MedSAM, (b) a comprehensive work flow in the inference time, and (c) a simplified diagram that illustrates the user interaction process.
  • Figure 3: Visualization of segmentation results for each method in different modalities, with Bbox illustrating varying degrees of rough approximations simulating expert annotations. (Col. 1) Images with an initial low-quality Bbox prompt; (Col. 2) SAM model; (Col. 3) MedSAM model; (Col. 4) Our model; (Col. 5) binary GT mask. Red: initial BBox, Blue: segmentation results Yellow: Dice Score.
  • Figure 4: Visualization of segmentation results for our method in different modalities. P 1 : low-quality box prompts, P 2 : refined box prompts, P 3 : refined box and point prompts, U 1 : step 1 Uncertainty Map, U 2 : step 2 Uncertainty Map, GT : ground truth. Green: initial Bbox, Red: refined prompts, Blue: segmentation results, Yellow: Dice Score