Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Curriculum Prompting Foundation Models for Medical Image Segmentation

Xiuqi Zheng, Yuhang Zhang, Haoran Zhang, Hongrui Liang, Xueqi Bao, Zhuqing Jiang, Qicheng Lao

TL;DR

The effectiveness of the proposed approach, which not only automates the prompt generation process but also yields superior performance compared to other SAM-based medical image segmentation methods, is demonstrated.

Abstract

Adapting large pre-trained foundation models, e.g., SAM, for medical image segmentation remains a significant challenge. A crucial step involves the formulation of a series of specialized prompts that incorporate specific clinical instructions. Past works have been heavily reliant on a singular type of prompt for each instance, necessitating manual input of an ideally correct prompt, which is less efficient. To tackle this issue, we propose to utilize prompts of different granularity, which are sourced from original images to provide a broader scope of clinical insights. However, combining prompts of varying types can pose a challenge due to potential conflicts. In response, we have designed a coarse-to-fine mechanism, referred to as curriculum prompting, that progressively integrates prompts of different types. Through extensive experiments on three public medical datasets across various modalities, we demonstrate the effectiveness of our proposed approach, which not only automates the prompt generation process but also yields superior performance compared to other SAM-based medical image segmentation methods. Code is available at: https://github.com/AnnaZzz-zxq/Curriculum-Prompting.

Curriculum Prompting Foundation Models for Medical Image Segmentation

TL;DR

The effectiveness of the proposed approach, which not only automates the prompt generation process but also yields superior performance compared to other SAM-based medical image segmentation methods, is demonstrated.

Abstract

Adapting large pre-trained foundation models, e.g., SAM, for medical image segmentation remains a significant challenge. A crucial step involves the formulation of a series of specialized prompts that incorporate specific clinical instructions. Past works have been heavily reliant on a singular type of prompt for each instance, necessitating manual input of an ideally correct prompt, which is less efficient. To tackle this issue, we propose to utilize prompts of different granularity, which are sourced from original images to provide a broader scope of clinical insights. However, combining prompts of varying types can pose a challenge due to potential conflicts. In response, we have designed a coarse-to-fine mechanism, referred to as curriculum prompting, that progressively integrates prompts of different types. Through extensive experiments on three public medical datasets across various modalities, we demonstrate the effectiveness of our proposed approach, which not only automates the prompt generation process but also yields superior performance compared to other SAM-based medical image segmentation methods. Code is available at: https://github.com/AnnaZzz-zxq/Curriculum-Prompting.
Paper Structure (16 sections, 6 equations, 3 figures, 3 tables)

This paper contains 16 sections, 6 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Overview
  4. Coarse Prompting
  5. Fine-grained Prompting
  6. Experiments and Results
  7. Dataset
  8. Experiment Settings and Metrics
  9. Results
  10. Our Proposed Approach Outperforms the Baselines on All Three Datasets.
  11. Visualization of Curriculum Prompting Process.
  12. Edge Points Served as Negative Prompts Can Better Improve SAM's Performance.
  13. Ablation Study.
  14. Training Time.
  15. Conclusion
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Overview of Curriculum Prompting: (a) Intermediate Prompt Generation, which prepares prompts for SAM; (b) Curriculum Prompting SAM, first utilizing self-generated box prompts to obtain coarse masks, and then acquire refined masks with self-generated point prompts and coarse masks (as mask prompts).
  • Figure 2: Qualitative comparisons between our curriculum prompting SAM and other segmentation methods on the TN3K dataset, including SOTA task-specific method TRFE+, and other SAM-based segmentation models.
  • Figure 3: The process of mask generation through our proposed curriculum prompting.