Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Segment Anything Is Not Always Perfect: An Investigation of SAM on Different Real-world Applications

Wei Ji, Jingjing Li, Qi Bi, Tingwei Liu, Wenbo Li, Li Cheng

TL;DR

This paper evaluates the Segment Anything Model (SAM) across a broad set of real-world segmentation tasks—natural images, agriculture, manufacturing, remote sensing, and healthcare—using qualitative visual analyses and quantitative MAE-based metrics. It shows that SAM generalizes well for common scenes and straightforward prompts, but struggles with fine-grained details, low-contrast scenes, small or irregular targets, and professional medical data, leading to notable performance gaps with state-of-the-art methods in several domains. The authors discuss practical limitations, such as foreground bias and reliance on domain knowledge, and propose future directions including domain-specific datasets, expanded prompting, multi-modal inputs, video-capable variants, and semi-supervised strategies to adapt SAM to specialized tasks. Overall, the work provides a structured assessment of SAM’s applicability beyond natural images and outlines concrete paths to improve robustness and domain adaptation for practical segmentation challenges.

Abstract

Recently, Meta AI Research approaches a general, promptable Segment Anything Model (SAM) pre-trained on an unprecedentedly large segmentation dataset (SA-1B). Without a doubt, the emergence of SAM will yield significant benefits for a wide array of practical image segmentation applications. In this study, we conduct a series of intriguing investigations into the performance of SAM across various applications, particularly in the fields of natural images, agriculture, manufacturing, remote sensing, and healthcare. We analyze and discuss the benefits and limitations of SAM, while also presenting an outlook on its future development in segmentation tasks. By doing so, we aim to give a comprehensive understanding of SAM's practical applications. This work is expected to provide insights that facilitate future research activities toward generic segmentation. Source code is publicly available.

Segment Anything Is Not Always Perfect: An Investigation of SAM on Different Real-world Applications

TL;DR

This paper evaluates the Segment Anything Model (SAM) across a broad set of real-world segmentation tasks—natural images, agriculture, manufacturing, remote sensing, and healthcare—using qualitative visual analyses and quantitative MAE-based metrics. It shows that SAM generalizes well for common scenes and straightforward prompts, but struggles with fine-grained details, low-contrast scenes, small or irregular targets, and professional medical data, leading to notable performance gaps with state-of-the-art methods in several domains. The authors discuss practical limitations, such as foreground bias and reliance on domain knowledge, and propose future directions including domain-specific datasets, expanded prompting, multi-modal inputs, video-capable variants, and semi-supervised strategies to adapt SAM to specialized tasks. Overall, the work provides a structured assessment of SAM’s applicability beyond natural images and outlines concrete paths to improve robustness and domain adaptation for practical segmentation challenges.

Abstract

Recently, Meta AI Research approaches a general, promptable Segment Anything Model (SAM) pre-trained on an unprecedentedly large segmentation dataset (SA-1B). Without a doubt, the emergence of SAM will yield significant benefits for a wide array of practical image segmentation applications. In this study, we conduct a series of intriguing investigations into the performance of SAM across various applications, particularly in the fields of natural images, agriculture, manufacturing, remote sensing, and healthcare. We analyze and discuss the benefits and limitations of SAM, while also presenting an outlook on its future development in segmentation tasks. By doing so, we aim to give a comprehensive understanding of SAM's practical applications. This work is expected to provide insights that facilitate future research activities toward generic segmentation. Source code is publicly available.
Paper Structure (23 sections, 12 figures, 1 table)

This paper contains 23 sections, 12 figures, 1 table.

Table of Contents

  1. Introduction
  2. Qualitative Investigation
  3. Natural Image Scenes
  4. Experiments on Different Subtasks
  5. Section Recap
  6. Agriculture
  7. Experiments on Different Subtasks
  8. Section Recap
  9. Manufacture
  10. Experiments on Different Subtasks
  11. Section Recap
  12. Remote Sensing
  13. Experiments on Different Subtasks
  14. Section Recap
  15. Healthcare
  16. ...and 8 more sections

Figures (12)

  • Figure 1: Results of Segment Anything Model (SAM SAM) on various real-world applications, where we adopt Everything mode to obtain SAM segmentations (right). The ground truth is masked with image for reference purpose (left). Best view is obtained when zooming in.
  • Figure 2: Application on salient object segmentation, where SAM$^{1/2/3}$ mean using Click, Box, and Everything modes respectively.
  • Figure 3: Application on camouflaged object segmentation, where SAM$^{1/2/3}$ mean using Click, Box, and Everything modes respectively.
  • Figure 4: Application on transparent object segmentation, where SAM$^{1/2/3}$ mean using Click, Box, and Everything modes respectively.
  • Figure 5: Application on shadow detection, where SAM$^{1/2/3}$ mean using Click, Box, and Everything modes respectively. The * indicates the SAM results within a box prompt.
  • ...and 7 more figures