Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Segmentation by registration-enabled SAM prompt engineering using five reference images

Yaxi Chen, Aleksandra Ivanova, Shaheer U. Saeed, Rikin Hargunani, Jie Huang, Chaozong Liu, Yipeng Hu

TL;DR

The paper addresses knee cartilage segmentation under limited annotation by introducing a registration-enabled SAM prompt-engineering framework that uses as few as five weakly-labelled reference images. It presents two pipelines, image-alignment and prompt-alignment, which align either the image or prompts to a reference set and then apply SAM with majority voting to produce final segmentations without requiring new image labels. Empirical results on post-surgery knee MRIs show strong bone segmentation (Dice up to ~0.89 for the femur and 0.87 for the tibia) and competitive cartilage segmentation (Dice ~0.53–0.52) compared to atlas-based methods and, for bone, approaching supervised nnUNet. The approach reduces annotation costs while enabling effective segmentation on out-of-distribution data, offering practical clinical utility and avenues for improvement through targeted registration and additional annotations.

Abstract

The recently proposed Segment Anything Model (SAM) is a general tool for image segmentation, but it requires additional adaptation and careful fine-tuning for medical image segmentation, especially for small, irregularly-shaped, and boundary-ambiguous anatomical structures such as the knee cartilage that is of interest in this work. Repaired cartilage, after certain surgical procedures, exhibits imaging patterns unseen to pre-training, posing further challenges for using models like SAM with or without general-purpose fine-tuning. To address this, we propose a novel registration-based prompt engineering framework for medical image segmentation using SAM. This approach utilises established image registration algorithms to align the new image (to-be-segmented) and a small number of reference images, without requiring segmentation labels. The spatial transformations generated by registration align either the new image or pre-defined point-based prompts, before using them as input to SAM. This strategy, requiring as few as five reference images with defined point prompts, effectively prompts SAM for inference on new images, without needing any segmentation labels. Evaluation of MR images from patients who received cartilage stem cell therapy yielded Dice scores of 0.89, 0.87, 0.53, and 0.52 for segmenting femur, tibia, femoral- and tibial cartilages, respectively. This outperforms atlas-based label fusion and is comparable to supervised nnUNet, an upper-bound fair baseline in this application, both of which require full segmentation labels for reference samples. The codes are available at: https://github.com/chrissyinreallife/KneeSegmentWithSAM.git

Segmentation by registration-enabled SAM prompt engineering using five reference images

TL;DR

The paper addresses knee cartilage segmentation under limited annotation by introducing a registration-enabled SAM prompt-engineering framework that uses as few as five weakly-labelled reference images. It presents two pipelines, image-alignment and prompt-alignment, which align either the image or prompts to a reference set and then apply SAM with majority voting to produce final segmentations without requiring new image labels. Empirical results on post-surgery knee MRIs show strong bone segmentation (Dice up to ~0.89 for the femur and 0.87 for the tibia) and competitive cartilage segmentation (Dice ~0.53–0.52) compared to atlas-based methods and, for bone, approaching supervised nnUNet. The approach reduces annotation costs while enabling effective segmentation on out-of-distribution data, offering practical clinical utility and avenues for improvement through targeted registration and additional annotations.

Abstract

The recently proposed Segment Anything Model (SAM) is a general tool for image segmentation, but it requires additional adaptation and careful fine-tuning for medical image segmentation, especially for small, irregularly-shaped, and boundary-ambiguous anatomical structures such as the knee cartilage that is of interest in this work. Repaired cartilage, after certain surgical procedures, exhibits imaging patterns unseen to pre-training, posing further challenges for using models like SAM with or without general-purpose fine-tuning. To address this, we propose a novel registration-based prompt engineering framework for medical image segmentation using SAM. This approach utilises established image registration algorithms to align the new image (to-be-segmented) and a small number of reference images, without requiring segmentation labels. The spatial transformations generated by registration align either the new image or pre-defined point-based prompts, before using them as input to SAM. This strategy, requiring as few as five reference images with defined point prompts, effectively prompts SAM for inference on new images, without needing any segmentation labels. Evaluation of MR images from patients who received cartilage stem cell therapy yielded Dice scores of 0.89, 0.87, 0.53, and 0.52 for segmenting femur, tibia, femoral- and tibial cartilages, respectively. This outperforms atlas-based label fusion and is comparable to supervised nnUNet, an upper-bound fair baseline in this application, both of which require full segmentation labels for reference samples. The codes are available at: https://github.com/chrissyinreallife/KneeSegmentWithSAM.git
Paper Structure (13 sections, 3 equations, 2 figures, 2 tables)

This paper contains 13 sections, 3 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Segment anything model
  4. Reference and segmentation samples
  5. Image registration
  6. Image alignment to the reference set
  7. Prompt alignment to the new sample
  8. Experiments and Results
  9. Dataset & Implementation Details
  10. Comparison and Ablation Experiments
  11. Results based on Clinically-Reported Evaluation Metrics
  12. Discussion and Conclusion
  13. Acknowledgement

Figures (2)

  • Figure 1: The proposed Image Alignment strategy, the first type of the two registration-enabled segmentation pipelines based on SAM prompt engineering, illustrated with samples of the reference images, point prompts and new images.
  • Figure 2: The proposed Prompt Alignment strategy, the second type of the two registration-enabled segmentation pipelines based on SAM prompt engineering, illustrated with samples of the new image, reference image and point prompts.