VFM-ISRefiner: Towards Better Adapting Vision Foundation Models for Interactive Segmentation of Remote Sensing Images

TL;DR

This work tackles interactive segmentation for remote sensing by integrating a frozen Vision Foundation Model backbone with a trainable adapter-based extractor and decoder, augmented by a hybrid convolution-Transformer attention and an improved probability map modulation to leverage historical user interactions. The method enables efficient domain adaptation to RS data while preserving general visual priors, achieving state-of-the-art performance in NoC85/NoC90 across multiple RS datasets with fewer user interactions. Extensive ablations confirm the effectiveness of the gated fusion, learnable theta, and the advanced modulation scheme, and cross-domain testing demonstrates robustness to different image modalities. The approach offers practical benefits for high-quality RS instance segmentation and scalable annotation workflows, with future work on more efficient fine-tuning and multimodal RS data integration.

Abstract

Interactive image segmentation(IIS) plays a critical role in generating precise annotations for remote sensing imagery, where objects often exhibit scale variations, irregular boundaries and complex backgrounds. However, existing IIS methods, primarily designed for natural images, struggle to generalize to remote sensing domains due to limited annotated data and computational overhead. To address these challenges, we proposed RS-ISRefiner, a novel click-based IIS framework tailored for remote sensing images. The framework employs an adapter-based tuning strategy that preserves the general representations of Vision Foundation Models while enabling efficient learning of remote sensing-specific spatial and boundary characteristics. A hybrid attention mechanism integrating convolutional local modeling with Transformer-based global reasoning enhances robustness against scale diversity and scene complexity. Furthermore, an improved probability map modulation scheme effectively incorporates historical user interactions, yielding more stable iterative refinement and higher boundary accuracy. Comprehensive experiments on six remote sensing datasets, including iSAID, ISPRS Potsdam, SandBar, NWPU, LoveDA Urban and WHUBuilding, demonstrate that RS-ISRefiner consistently outperforms state-of-the-art IIS methods in terms of segmentation accuracy, efficiency and interaction cost. These results confirm the effectiveness and generalizability of our framework, making it highly suitable for high-quality instance segmentation in practical remote sensing scenarios. The codes are available at https://github.com/wondelyan/VFM-ISRefiner .

Figures (11)

• Figure 1: Structure Comparison. (a) represents the common IIS structure. (b) is the simplified structure used in our paper.
• Figure 2: Overall Structure. Our architecture consists of three main components: a frozen Vision Foundation Model (VFM) backbone (blue region), a parameter-trained extractor and decoder (orange-red region) and a probability map modulation module at the top of the diagram. Both the training and testing processes follow a recursive workflow.
• Figure 3: Previous Mask Processing Module. The module is mainly composed of four convolutional modules and one Xception-convolutional module, where the light orange region at the bottom shows the detailed structure of the Xception-convolutional module.
• Figure 4: High Quality Feature Extractor. This module comprises an image feature extractor (light-colored region in the middle) and a feature fusion module (green region on the right). The image feature extractor includes one ResBlock, three DeformableResBlocks and one 1×1 convolution, with the detailed architectures of ResBlock and Deformable ResBlock shown on the left of the diagram. The feature fusion module is primarily composed of cross-attention blocks, where $\theta$ serves as a learnable parameter.
• Figure 5: Transformer Decoder. The module consists of a Decoder Layer (the yellow region on the left) and a dual-branch structure (the right region), where the latter includes a Token path and a Feature path. Their outputs are fused via a dynamic convolution module to generate the predicted mask.