SAM2-Adapter: Evaluating & Adapting Segment Anything 2 in Downstream Tasks: Camouflage, Shadow, Medical Image Segmentation, and More

TL;DR

Problem: SAM2 struggles on low-level segmentation tasks such as camouflaged object detection, shadow detection, and medical imagery. Approach: SAM2-Adapter introduces a multi-adapter configuration that injects task-specific prompts into SAM2's hierarchical transformer backbone, enabling task-aware adaptation without altering the core encoder. Contributions: first adapter-based method for SAM2 that attains new state-of-the-art results on CAMO/COD10K/CHAMELEON, ISTD, and kvasir-SEG, with ablation analyses confirming the benefit of four-stage adapters and the SAM2 backbone. Significance: offers a practical, generalizable recipe to adapt large segmentation models to diverse domains, with code and pretrained resources released for community use.

Abstract

The advent of large models, also known as foundation models, has significantly transformed the AI research landscape, with models like Segment Anything (SAM) achieving notable success in diverse image segmentation scenarios. Despite its advancements, SAM encountered limitations in handling some complex low-level segmentation tasks like camouflaged object and medical imaging. In response, in 2023, we introduced SAM-Adapter, which demonstrated improved performance on these challenging tasks. Now, with the release of Segment Anything 2 (SAM2), a successor with enhanced architecture and a larger training corpus, we reassess these challenges. This paper introduces SAM2-Adapter, the first adapter designed to overcome the persistent limitations observed in SAM2 and achieve new state-of-the-art (SOTA) results in specific downstream tasks including medical image segmentation, camouflaged (concealed) object detection, and shadow detection. SAM2-Adapter builds on the SAM-Adapter's strengths, offering enhanced generalizability and composability for diverse applications. We present extensive experimental results demonstrating SAM2-Adapter's effectiveness. We show the potential and encourage the research community to leverage the SAM2 model with our SAM2-Adapter for achieving superior segmentation outcomes. Code, pre-trained models, and data processing protocols are available at http://tianrun-chen.github.io/SAM-Adaptor/

Figures (6)

• Figure 1: The architecture of the proposed SAM2-Adapter. We show the differences between SAM2-Adapter and SAM-Adapter in the Ablation Study section.
• Figure 2: Visualization for Camouflaged Image Segmentation in CHAMELEON dataset As shown in the figure, SAM often fails to detect animals that are visually camouflaged within their natural environments and can sometimes produce irrelevant results. SAM2 also struggles with similar issues and produces non-meaningful outcomes. However, by incorporating SAM-Adapter, our approach significantly improves object segmentation performance. Furthermore, SAM2-Adapter demonstrates even better performance than SAM-Adapter. The samples depicted are from the CHAMELEON dataset.
• Figure 3: Camouflaged Image Segmentation in COD-10K dataset As shown in the figure, SAM struggles to detect animals that are visually camouflaged within their natural environments and can sometimes produce results that lack meaningful segmentation. SAM2 also faces similar challenges, often resulting in no output or false results. However, by incorporating SAM2-Adapter, our method significantly improves object segmentation performance, surpassing SAM-Adapter.
• Figure 4: Camouflaged Segmentation of CAMO dataset. The SAM and SAM 2 failed to perceive those animals that are visually ‘hidden’/concealed in their natural surroundings. The effectiveness of SAM2-Adapter is further validated in the visualized results.
• Figure 5: Shadow Detection Visualized. Both SAM and SAM2 have no understanding of the "shadow" concept without proper prompting. They produce meaningless results. SAM-Adapter and SAM2-Adapter perform equally well in shadow detection tasks.