Referring Remote Sensing Image Segmentation via Bidirectional Alignment Guided Joint Prediction

TL;DR

This work tackles Referring Remote Sensing Image Segmentation (RRSIS) by addressing the vision–language gap in high‑resolution RS imagery and the challenge of small, ambiguous targets. It introduces a three‑pillar framework—Bidirectional Spatial Correlation (BSC) for cross‑modal alignment, a Target‑Background TwinStream Decoder (T‑BTD) for joint foreground/background prediction, and a Dual‑Modal Object Learning Strategy (D‑MOLS) with a Reconstruction objective—to achieve precise, robust segmentation. Across RefSegRS and RRSIS‑D, the approach yields state‑of‑the‑art results, with notable gains on fine‑grained and multi‑scale targets and improved boundary delineation. The method demonstrates strong practical potential for ecological monitoring, urban planning, and disaster response, while pointing to future directions such as incorporating large language models to further bridge language understanding with RS imagery.

Abstract

Referring Remote Sensing Image Segmentation (RRSIS) is critical for ecological monitoring, urban planning, and disaster management, requiring precise segmentation of objects in remote sensing imagery guided by textual descriptions. This task is uniquely challenging due to the considerable vision-language gap, the high spatial resolution and broad coverage of remote sensing imagery with diverse categories and small targets, and the presence of clustered, unclear targets with blurred edges. To tackle these issues, we propose \ours, a novel framework designed to bridge the vision-language gap, enhance multi-scale feature interaction, and improve fine-grained object differentiation. Specifically, \ours introduces: (1) the Bidirectional Spatial Correlation (BSC) for improved vision-language feature alignment, (2) the Target-Background TwinStream Decoder (T-BTD) for precise distinction between targets and non-targets, and (3) the Dual-Modal Object Learning Strategy (D-MOLS) for robust multimodal feature reconstruction. Extensive experiments on the benchmark datasets RefSegRS and RRSIS-D demonstrate that \ours achieves state-of-the-art performance. Specifically, \ours improves the overall IoU (oIoU) by 3.76 percentage points (80.57) and 1.44 percentage points (79.23) on the two datasets, respectively. Additionally, it outperforms previous methods in the mean IoU (mIoU) by 5.37 percentage points (67.95) and 1.84 percentage points (66.04), effectively addressing the core challenges of RRSIS with enhanced precision and robustness.

Figures (10)

• Figure 1: The illustration of the pipeline comparison between existing methods and our design for the RRSIS task.
• Figure 2: A comparison of two scenarios. (a) shows clear targets with simple descriptions, demonstrating the superior performance of existing methods. (b) illustrates complex descriptions with ambiguous backgrounds, highlighting the limitations of current approaches.
• Figure 3: The overall framework of the proposed BTDNet Network includes the following components: (a)Bidirectional Feature Extractor, in which the visual and text encoders extract features from image and text inputs, respectively, with the Bidirectional Spatial Correlation Module enabling bidirectional information exchange at the feature level. (b)Target-Background TwinStream, implementing a text-aware dual-stream inference strategy for entity targets and category-agnostic areas during the mask prediction stage. (c)Dual-Modal Object Learning, designed to apply a reconstruction loss named Lre after erasing text entity words, in which the reconstruction module leverages multimodal visual features to guide text reconstruction.
• Figure 4: The illustration of Bidirectional Spatial Correlation Module. The correlation of information between image-to-text and text-to-image will be performed based on spatial context.
• Figure 5: The illustration of the Target-Background TwinStream Decoder, comprising the Multi-scale Context Integration for multi-scale feature enhancement and the Learnable Fg-Bg Predictor for distinguishing target and background regions.