Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MIAS-SAM: Medical Image Anomaly Segmentation without thresholding

Marco Colussi, Dragan Ahmetovic, Sergio Mascetti

TL;DR

MIAS-SAM addresses unsupervised medical anomaly segmentation without thresholding by constructing a patch-based memory bank of normal features from a SAM ViT encoder and scoring test patches via memory-distance. It computes the anomaly map's center of gravity and uses the resulting coordinates as a point prompt to guide the SAM decoder, yielding accurate, threshold-free segmentation. Across Brain MRI, Liver CT, and Retina OCT, MIAS-SAM achieves competitive to state-of-the-art DICE scores and excels on Retina OCT, while avoiding dataset-specific textual prompts. The approach highlights the potential of memory-augmented embeddings and gravity-based prompting for robust medical anomaly segmentation using foundational models.

Abstract

This paper presents MIAS-SAM, a novel approach for the segmentation of anomalous regions in medical images. MIAS-SAM uses a patch-based memory bank to store relevant image features, which are extracted from normal data using the SAM encoder. At inference time, the embedding patches extracted from the SAM encoder are compared with those in the memory bank to obtain the anomaly map. Finally, MIAS-SAM computes the center of gravity of the anomaly map to prompt the SAM decoder, obtaining an accurate segmentation from the previously extracted features. Differently from prior works, MIAS-SAM does not require to define a threshold value to obtain the segmentation from the anomaly map. Experimental results conducted on three publicly available datasets, each with a different imaging modality (Brain MRI, Liver CT, and Retina OCT) show accurate anomaly segmentation capabilities measured using DICE score. The code is available at: https://github.com/warpcut/MIAS-SAM

MIAS-SAM: Medical Image Anomaly Segmentation without thresholding

TL;DR

MIAS-SAM addresses unsupervised medical anomaly segmentation without thresholding by constructing a patch-based memory bank of normal features from a SAM ViT encoder and scoring test patches via memory-distance. It computes the anomaly map's center of gravity and uses the resulting coordinates as a point prompt to guide the SAM decoder, yielding accurate, threshold-free segmentation. Across Brain MRI, Liver CT, and Retina OCT, MIAS-SAM achieves competitive to state-of-the-art DICE scores and excels on Retina OCT, while avoiding dataset-specific textual prompts. The approach highlights the potential of memory-augmented embeddings and gravity-based prompting for robust medical anomaly segmentation using foundational models.

Abstract

This paper presents MIAS-SAM, a novel approach for the segmentation of anomalous regions in medical images. MIAS-SAM uses a patch-based memory bank to store relevant image features, which are extracted from normal data using the SAM encoder. At inference time, the embedding patches extracted from the SAM encoder are compared with those in the memory bank to obtain the anomaly map. Finally, MIAS-SAM computes the center of gravity of the anomaly map to prompt the SAM decoder, obtaining an accurate segmentation from the previously extracted features. Differently from prior works, MIAS-SAM does not require to define a threshold value to obtain the segmentation from the anomaly map. Experimental results conducted on three publicly available datasets, each with a different imaging modality (Brain MRI, Liver CT, and Retina OCT) show accurate anomaly segmentation capabilities measured using DICE score. The code is available at: https://github.com/warpcut/MIAS-SAM

Paper Structure

This paper contains 19 sections, 2 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Embedding based.
  4. Foundational models.
  5. Method: MIAS-SAM
  6. Problem formulation
  7. Embedding extraction phase
  8. Anomaly segmentation phase
  9. Experimental evaluation
  10. Evaluation methodology
  11. Implementation details.
  12. Baselines.
  13. Evaluation metrics.
  14. Datasets
  15. Results
  16. ...and 4 more sections

Figures (2)

  • Figure 1: Overview of the MIAS-SAM methodology. First, image embeddings are extracted using the SAM encoder and patches are extracted and stored in a memory bank. At test time, images are encoded, and anomaly map are generated based on patch-wise distances. The anomaly maps are then used to compute a spatial prompts, which guide the decoder in segmenting the anomalous region.
  • Figure 2: Qualitative results of MIAS-SAM. Each row corresponds to a different dataset: brain (top), liver (middle), and retina (bottom). The first column shows the input image, the second the generated anomaly map, the third the first segmentation mask, and the fourth the third segmentation mask, that best captures the anomaly.