Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Putting the Segment Anything Model to the Test with 3D Knee MRI - A Comparison with State-of-the-Art Performance

Oliver Mills, Philip Conaghan, Nishant Ravikumar, Samuel Relton

TL;DR

This study evaluates the Segment Anything Model (SAM), a foundation vision transformer, for fully automated segmentation of knee menisci in 3D MRI and compares it to a 3D U-Net baseline. SAM is examined in two training regimes: decoder-only fine-tuning and end-to-end training, using 2D sagittal slices lifted to 3D masks. While end-to-end SAM achieves a Dice score of $0.87\pm0.03$, matching the 3D U-Net in Dice, its Hausdorff distance underperforms, indicating less accurate boundary localization and potential limitations for deriving OA biomarkers. The results suggest that, for fine anatomical structures with low contrast, a standard 3D U-Net still offers superior 3D morphology preservation, and SAM’s generalizability may not yet extend to such 3D medical segmentation tasks. Future work should explore longitudinal analyses and larger cohorts to uncover potential OA biomarkers from automated meniscus segmentation.

Abstract

Menisci are cartilaginous tissue found within the knee that contribute to joint lubrication and weight dispersal. Damage to menisci can lead to onset and progression of knee osteoarthritis (OA), a condition that is a leading cause of disability, and for which there are few effective therapies. Accurate automated segmentation of menisci would allow for earlier detection and treatment of meniscal abnormalities, as well as shedding more light on the role the menisci play in OA pathogenesis. Focus in this area has mainly used variants of convolutional networks, but there has been no attempt to utilise recent large vision transformer segmentation models. The Segment Anything Model (SAM) is a so-called foundation segmentation model, which has been found useful across a range of different tasks due to the large volume of data used for training the model. In this study, SAM was adapted to perform fully-automated segmentation of menisci from 3D knee magnetic resonance images. A 3D U-Net was also trained as a baseline. It was found that, when fine-tuning only the decoder, SAM was unable to compete with 3D U-Net, achieving a Dice score of $0.81\pm0.03$, compared to $0.87\pm0.03$, on a held-out test set. When fine-tuning SAM end-to-end, a Dice score of $0.87\pm0.03$ was achieved. The performance of both the end-to-end trained SAM configuration and the 3D U-Net were comparable to the winning Dice score ($0.88\pm0.03$) in the IWOAI Knee MRI Segmentation Challenge 2019. Performance in terms of the Hausdorff Distance showed that both configurations of SAM were inferior to 3D U-Net in matching the meniscus morphology. Results demonstrated that, despite its generalisability, SAM was unable to outperform a basic 3D U-Net in meniscus segmentation, and may not be suitable for similar 3D medical image segmentation tasks also involving fine anatomical structures with low contrast and poorly-defined boundaries.

Putting the Segment Anything Model to the Test with 3D Knee MRI - A Comparison with State-of-the-Art Performance

TL;DR

This study evaluates the Segment Anything Model (SAM), a foundation vision transformer, for fully automated segmentation of knee menisci in 3D MRI and compares it to a 3D U-Net baseline. SAM is examined in two training regimes: decoder-only fine-tuning and end-to-end training, using 2D sagittal slices lifted to 3D masks. While end-to-end SAM achieves a Dice score of , matching the 3D U-Net in Dice, its Hausdorff distance underperforms, indicating less accurate boundary localization and potential limitations for deriving OA biomarkers. The results suggest that, for fine anatomical structures with low contrast, a standard 3D U-Net still offers superior 3D morphology preservation, and SAM’s generalizability may not yet extend to such 3D medical segmentation tasks. Future work should explore longitudinal analyses and larger cohorts to uncover potential OA biomarkers from automated meniscus segmentation.

Abstract

Menisci are cartilaginous tissue found within the knee that contribute to joint lubrication and weight dispersal. Damage to menisci can lead to onset and progression of knee osteoarthritis (OA), a condition that is a leading cause of disability, and for which there are few effective therapies. Accurate automated segmentation of menisci would allow for earlier detection and treatment of meniscal abnormalities, as well as shedding more light on the role the menisci play in OA pathogenesis. Focus in this area has mainly used variants of convolutional networks, but there has been no attempt to utilise recent large vision transformer segmentation models. The Segment Anything Model (SAM) is a so-called foundation segmentation model, which has been found useful across a range of different tasks due to the large volume of data used for training the model. In this study, SAM was adapted to perform fully-automated segmentation of menisci from 3D knee magnetic resonance images. A 3D U-Net was also trained as a baseline. It was found that, when fine-tuning only the decoder, SAM was unable to compete with 3D U-Net, achieving a Dice score of , compared to , on a held-out test set. When fine-tuning SAM end-to-end, a Dice score of was achieved. The performance of both the end-to-end trained SAM configuration and the 3D U-Net were comparable to the winning Dice score () in the IWOAI Knee MRI Segmentation Challenge 2019. Performance in terms of the Hausdorff Distance showed that both configurations of SAM were inferior to 3D U-Net in matching the meniscus morphology. Results demonstrated that, despite its generalisability, SAM was unable to outperform a basic 3D U-Net in meniscus segmentation, and may not be suitable for similar 3D medical image segmentation tasks also involving fine anatomical structures with low contrast and poorly-defined boundaries.

Paper Structure

This paper contains 16 sections, 1 equation, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Deep Learning in Medical Image Analysis
  3. Materials and Methods
  4. Data
  5. Preprocessing
  6. 3D U-Net
  7. Segment Anything Model
  8. Evaluation Metrics
  9. Dice Score
  10. Hausdorff Distance
  11. Average Transverse Thickness
  12. Model Training
  13. Results and Discussion
  14. Case Analyses
  15. Conclusion
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Preprocessing steps performed on the MR Images before model training. Windowing was performed between 0 and 0.005. The cropped region was selected based on the variation in location of ground truths in the train and validation sets.
  • Figure 2: Violin plots showing the distributions of the dice score (a) and Hausdorff distance (b) of the three model configurations when predicting on the test set. In the violin interior, a box plot is shown. In SAM 1, only the mask decoder was trained. In SAM 2, the model was trained end-to-end.
  • Figure 3: Bland-Altman plots showing the difference in transverse thickness between masks generated by SAM 2 (a) and 3D U-Net (b). The difference was calculated by subtracting the ground truth thickness from the generated mask thickness.
  • Figure 4: Two atypical examples from the test dataset that visually compare the masks predicted by the segmentation models investigated with the ground truth. In (a), a ground truth mask is shown where the medial meniscus was separated into two (white arrow). (b-d) are the generated masks from SAM 1, SAM 2, and 3D U-Net respectively. (e) shows a test case with what appears to be a partial meniscectomy on the posterior medial meniscal horn (white arrow). (f-h) are again the generated masks from SAM 1, SAM 2, and 3D U-Net respectively. All images in this figure were created by summing the 3D masks through the inferior-superior axis of the body, as if looking down on the mask from above. The brighter a pixel, the thicker the meniscus in this transverse position.
  • Figure 5: Surface mesh representations of the worst-performing (Dice score) predicted masks of 3D U-Net and end-to-end trained SAM (SAM 2) from the test knee MRI set, shown alongside the corresponding ground truth. The meshes contain both the lateral and medial menisci, which are seen on the left and right of each subfigure.