Scale-wise Bidirectional Alignment Network for Referring Remote Sensing Image Segmentation

TL;DR

This paper tackles pixel-wise referring segmentation in remote sensing imagery (RRSIS), addressing the limitations of one-directional language-guided refinement and the challenge of multi-scale targets. It introduces SBANet, a framework built on Swin Transformer and BERT that replaces the traditional pixel-word attention with a Bidirectional Alignment Module (BAM) that updates both visual and linguistic features via learnable query tokens and query-text alignment, complemented by a dynamic feature selection block. A text-conditioned channel and spatial aggregator (TSCA) further enables cross-scale information exchange by applying text-guided channel and spatial attentions before decoding. Extensive experiments on RRSIS-D and RefSegRS demonstrate SBANet's superior performance over state-of-the-art methods, with ablations confirming the contributions of BAM, DFS, and TSCA to improved cross-modal interaction and scale-aware reasoning. The work advances practical V&L reasoning for high-resolution aerial imagery, enabling more accurate and context-aware pixel-level segmentation conditioned on natural language prompts."

Abstract

The goal of referring remote sensing image segmentation (RRSIS) is to extract specific pixel-level regions within an aerial image via a natural language expression. Recent advancements, particularly Transformer-based fusion designs, have demonstrated remarkable progress in this domain. However, existing methods primarily focus on refining visual features using language-aware guidance during the cross-modal fusion stage, neglecting the complementary vision-to-language flow. This limitation often leads to irrelevant or suboptimal representations. In addition, the diverse spatial scales of ground objects in aerial images pose significant challenges to the visual perception capabilities of existing models when conditioned on textual inputs. In this paper, we propose an innovative framework called Scale-wise Bidirectional Alignment Network (SBANet) to address these challenges for RRSIS. Specifically, we design a Bidirectional Alignment Module (BAM) with learnable query tokens to selectively and effectively represent visual and linguistic features, emphasizing regions associated with key tokens. BAM is further enhanced with a dynamic feature selection block, designed to provide both macro- and micro-level visual features, preserving global context and local details to facilitate more effective cross-modal interaction. Furthermore, SBANet incorporates a text-conditioned channel and spatial aggregator to bridge the gap between the encoder and decoder, enhancing cross-scale information exchange in complex aerial scenarios. Extensive experiments demonstrate that our proposed method achieves superior performance in comparison to previous state-of-the-art methods on the RRSIS-D and RefSegRS datasets, both quantitatively and qualitatively. The code will be released after publication.

Figures (8)

• Figure 1: Illustration of different methods for RRSIS and their corresponding results on two examples from the RRSIS-D liu2024rmsin dataset. In (a), the previous methods utilize language-aware transformers to only update visual features, while our method selectively refines both visual and linguistic features with vision-to-language and language-to-vision flows in (b). To distinguish the inner guidance in the cross-modal alignment modules, we represent the directional flows with dashed arrows in orange and blue. We further present two examples sharing the same expression: (c) and (e) are results obtained using (a), while (d) and (f) are results predicted by our method.
• Figure 2: The overview of SBANet framework. We utilize the base Swin Transformer liu2021swin and BERT devlin2018bert as visual and textual encoders for extracting visual and linguistic features, respectively. At the first stage, the visual features $F_{V_1}$ and original linguistic features $F_{L}$ are fed into the proposed bidirectional alignment module (BAM) to obtain updated results $F'_{V_1}$ and $F'_{L_1}$. We illustrate the different directional flows using dashed arrows in orange and blue. Then $F'_{V_i}$ are employed through a language gate (on the upper part) to generate $F'_{V_{i+1}}$ at the next stage while $F'_{L_1}$ are used as input for the next BAM. With the updated visual and linguistic features together, we introduce a text-conditioned channel and spatial aggregator to enhance cross-scale information exchange before decoding the final mask. For brevity, we do not show the concatenation for the decoder.
• Figure 3: The architecture of the proposed BAM. The learnable query tokens enable sparsely represent the visual context with position embedding (not shown here for a clear presentation), and they are then used to guide the update of linguistic features through a query-text token alignment module. A dynamic feature selection block processes pyramid visual features for a selective pixel-word alignment with $F'_{L_i}$, subsequently obtaining updated visual features $F'_{V_i}$ with upsampling and MLP.
• Figure 4: The architecture of the proposed text-conditioned channel and spatial aggregator (TSCA). The shapes of queries, keys and values are also present, where $m$, $c_i$, and $c_c$ denote the number of patches, the number of channels at each scale and the number of all concatenated features, respectively. The updated linguistic feature are incorporated with multi-scale visual features through channel and spatial attentions before decoding the final mask. It produces enhanced multi-scale representations and connects between the multi-modal encoder and visual decoder.
• Figure 5: Qualitative comparison of different methods on the RRSIS-D liu2024rmsin dataset (Best view in Zoom).