SU-SAM: A Simple Unified Framework for Adapting Segment Anything Model in Underperformed Scenes

TL;DR

This work addresses SAM's limited performance on specialized data by introducing SU-SAM, a simple unified PEFT framework that tunes SAM without task-specific designs. Adaptation is modeled as learning a residual $\Delta \boldsymbol{h}$ via Adapter/LoRA modules inserted in Transformer blocks, with four variants (Series, Parallel, Mixed, LoRA) to explore insertion strategies. Extensive experiments across nine datasets and six tasks demonstrate competitive or superior accuracy relative to state-of-the-art, and a generalized multi-dataset SU-SAM model shows strong cross-task generalization. The approach is lightweight, data-agnostic, and easily extendable to SAM-based and Transformer-based architectures, offering practical benefits for broad deployment across diverse segmentation problems.

Abstract

Segment anything model (SAM) has demonstrated excellent generalizability in common vision scenarios, yet falling short of the ability to understand specialized data. Recently, several methods have combined parameter-efficient techniques with task-specific designs to fine-tune SAM on particular tasks. However, these methods heavily rely on handcraft, complicated, and task-specific designs, and pre/post-processing to achieve acceptable performances on downstream tasks. As a result, this severely restricts generalizability to other downstream tasks. To address this issue, we present a simple and unified framework, namely SU-SAM, that can easily and efficiently fine-tune the SAM model with parameter-efficient techniques while maintaining excellent generalizability toward various downstream tasks. SU-SAM does not require any task-specific designs and aims to improve the adaptability of SAM-like models significantly toward underperformed scenes. Concretely, we abstract parameter-efficient modules of different methods into basic design elements in our framework. Besides, we propose four variants of SU-SAM, i.e., series, parallel, mixed, and LoRA structures. Comprehensive experiments on nine datasets and six downstream tasks to verify the effectiveness of SU-SAM, including medical image segmentation, camouflage object detection, salient object segmentation, surface defect segmentation, complex object shapes, and shadow masking. Our experimental results demonstrate that SU-SAM achieves competitive or superior accuracy compared to state-of-the-art methods. Furthermore, we provide in-depth analyses highlighting the effectiveness of different parameter-efficient designs within SU-SAM. In addition, we propose a generalized model and benchmark, showcasing SU-SAM's generalizability across all diverse datasets simultaneously.

Figures (6)

• Figure 1: Top: contrast between prior SAM-based Paramter-efficient fine-tuning (PEFT) methods wu2023medicalzhang2023customizedchen2023sam and our SU-SAM. Previous works combined task-specific designs to pursue higher accuracy. Unlike them, our SU-SAM removes all task-related operations. Bottom: extensive experiments on nine datasets covering six tasks demonstrate that the proposed SU-SAM can significantly improve the performance of SAM. The evaluation metric is MAE, and a lower value indicates better performance.
• Figure 2: Illustration of our presented SU-SAM framework, a simple and unified framework for adapting SAM in underpeformed scenes. SU-SAM brings slight trainable parameters, e.g., adapters and LoRA, to different positions of the Transformer backbone in various manners.
• Figure 3: Illustration of the model architectures of four different variants, i.e., Series SU-SAM, Parallel SU-SAM, Mixed SU-SAM, LoRA SU-SAM. In this figure, some SAM modules are omitted, and we solely focus on modified components.
• Figure 4: Visualization results. We randomly selected images from the test sets of several datasets (COD10K COD, SBU SBU, DUTS DUTS, CHAMELEON skurowski2018animal, CAMO le2019anabranch).SU-SAM significantly improves the performance of the original SAM.
• Figure 5: Visualization results on camouflaged object segmentation, surface defect segmentation, and complex object shapes segmentation.