Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Novel adaptation of video segmentation to 3D MRI: efficient zero-shot knee segmentation with SAM2

Andrew Seohwan Yu, Mohsen Hariri, Xuecen Zhang, Mingrui Yang, Vipin Chaudhary, Xiaojuan Li

TL;DR

This work tackles domain transfer in medical image segmentation by adapting Segment Anything Model 2 (SAM2) to 3D MRI. By treating 3D knee volumes as sequences of video frames, the authors leverage SAM2's memory and video prediction capabilities to achieve zero-shot, single-prompt segmentation without training. On the OAI-ZIB knee dataset, SAM2 demonstrates strong performance, with a best 3-points+video configuration yielding a median 3D DSC of 0.9196 and tibia DSCs near 0.9643 in some setups, indicating effective cross-slice prompt propagation. The study suggests a scalable, cost-effective approach to automated knee bone segmentation, with potential for broader clinical deployment and extension to other imaging modalities.

Abstract

Intelligent medical image segmentation methods are rapidly evolving and being increasingly applied, yet they face the challenge of domain transfer, where algorithm performance degrades due to different data distributions between source and target domains. To address this, we introduce a method for zero-shot, single-prompt segmentation of 3D knee MRI by adapting Segment Anything Model 2 (SAM2), a general-purpose segmentation model designed to accept prompts and retain memory across frames of a video. By treating slices from 3D medical volumes as individual video frames, we leverage SAM2's advanced capabilities to generate motion- and spatially-aware predictions. We demonstrate that SAM2 can efficiently perform segmentation tasks in a zero-shot manner with no additional training or fine-tuning, accurately delineating structures in knee MRI scans using only a single prompt. Our experiments on the Osteoarthritis Initiative Zuse Institute Berlin (OAI-ZIB) dataset reveal that SAM2 achieves high accuracy on 3D knee bone segmentation, with a testing Dice similarity coefficient of 0.9643 on tibia. We also present results generated using different SAM2 model sizes, different prompt schemes, as well as comparative results from the SAM1 model deployed on the same dataset. This breakthrough has the potential to revolutionize medical image analysis by providing a scalable, cost-effective solution for automated segmentation, paving the way for broader clinical applications and streamlined workflows.

Novel adaptation of video segmentation to 3D MRI: efficient zero-shot knee segmentation with SAM2

TL;DR

This work tackles domain transfer in medical image segmentation by adapting Segment Anything Model 2 (SAM2) to 3D MRI. By treating 3D knee volumes as sequences of video frames, the authors leverage SAM2's memory and video prediction capabilities to achieve zero-shot, single-prompt segmentation without training. On the OAI-ZIB knee dataset, SAM2 demonstrates strong performance, with a best 3-points+video configuration yielding a median 3D DSC of 0.9196 and tibia DSCs near 0.9643 in some setups, indicating effective cross-slice prompt propagation. The study suggests a scalable, cost-effective approach to automated knee bone segmentation, with potential for broader clinical deployment and extension to other imaging modalities.

Abstract

Intelligent medical image segmentation methods are rapidly evolving and being increasingly applied, yet they face the challenge of domain transfer, where algorithm performance degrades due to different data distributions between source and target domains. To address this, we introduce a method for zero-shot, single-prompt segmentation of 3D knee MRI by adapting Segment Anything Model 2 (SAM2), a general-purpose segmentation model designed to accept prompts and retain memory across frames of a video. By treating slices from 3D medical volumes as individual video frames, we leverage SAM2's advanced capabilities to generate motion- and spatially-aware predictions. We demonstrate that SAM2 can efficiently perform segmentation tasks in a zero-shot manner with no additional training or fine-tuning, accurately delineating structures in knee MRI scans using only a single prompt. Our experiments on the Osteoarthritis Initiative Zuse Institute Berlin (OAI-ZIB) dataset reveal that SAM2 achieves high accuracy on 3D knee bone segmentation, with a testing Dice similarity coefficient of 0.9643 on tibia. We also present results generated using different SAM2 model sizes, different prompt schemes, as well as comparative results from the SAM1 model deployed on the same dataset. This breakthrough has the potential to revolutionize medical image analysis by providing a scalable, cost-effective solution for automated segmentation, paving the way for broader clinical applications and streamlined workflows.
Paper Structure (5 sections, 2 figures, 1 table)

This paper contains 5 sections, 2 figures, 1 table.

Table of Contents

  1. Description of purpose
  2. METHODS
  3. RESULTS
  4. New or breakthrough work to be presented
  5. CONCLUSIONS

Figures (2)

  • Figure 1: Adaptation of the SAM2 model's video predictor to a 3D MR volume in two stages: one slice from the input MRI volume is segmented using the image predictor; then the video processor and memory component propagates the prompts and the predictions to the rest of the volume iteratively.
  • Figure 2: Different prompts (white) provided to SAM1 and SAM2, with femur segmentation predictions overlaid (red). Single point prompt fails to limit the extent of the prediction to the femur (left); bounding box (center) and negative prompts (right) successfully limit the extent.