Tuning a SAM-Based Model with Multi-Cognitive Visual Adapter to Remote Sensing Instance Segmentation

TL;DR

The paper tackles adapting a foundation segmentation model to remote sensing, where pretraining data and prompt-based interaction limit automatic instance segmentation. It introduces MC-SAM SEG, which integrates a SAM-Mona encoder with Mona adapters and a lightweight feature aggregator, followed by a pixel decoder and Transformer decoder to produce prompt-free masks and class labels. Key results show MC-SAM SEG achieving $AP_{mask}$ of 71.2% on WHU and 66.4% on HRSID, outperforming RSPrompter and SAM SEG, thereby validating Mona-tuning as an effective domain adaptation strategy for remote sensing. The approach offers efficient transfer learning via PEFT and enables robust, automatic segmentation of large-scale optical and SAR remote sensing data with strong generalization.

Abstract

The Segment Anything Model (SAM), a foundational model designed for promptable segmentation tasks, demonstrates exceptional generalization capabilities, making it highly promising for natural scene image segmentation. However, SAM's lack of pretraining on massive remote sensing images and its interactive structure limit its automatic mask prediction capabilities. In this paper, a Multi-Cognitive SAM-Based Instance Segmentation Model (MC-SAM SEG) is introduced to employ SAM on remote sensing domain. The SAM-Mona encoder utilizing the Multi-cognitive Visual Adapter (Mona) is conducted to facilitate SAM's transfer learning in remote sensing applications. The proposed method named MC-SAM SEG extracts high-quality features by fine-tuning the SAM-Mona encoder along with a feature aggregator. Subsequently, a pixel decoder and transformer decoder are designed for prompt-free mask generation and instance classification. The comprehensive experiments are conducted on the HRSID and WHU datasets for instance segmentation tasks on Synthetic Aperture Radar (SAR) images and optical remote sensing images respectively. The evaluation results indicate the proposed method surpasses other deep learning algorithms and verify its effectiveness and generalization.

Figures (7)

• Figure 1: Overview network architecture of MC-SAM SEG, which employs an encoder-decoder architecture.
• Figure 2: (a) Standard Transformer block. (b) SAM-Mona encoder's Transformer block.(c) Architecture of Mona Adapter.
• Figure 3: The structure of SAM-Mona image encoder.
• Figure 4: The structure of MC-SAM SEG's mask decoder.
• Figure 5: Transfer Learning: The blue modules represent frozen components, with non-trainable parameters. The orange modules represent trainable components, where parameters can be fine-tuned. The gray modules indicate partially trainable components.