Understanding What Is Not Said:Referring Remote Sensing Image Segmentation with Scarce Expressions

TL;DR

This work addresses the scarcity of high-quality referring expressions in remote sensing image segmentation by introducing Weakly Referring Expression Learning (WREL-RRSIS), which couples abundant class-name cues with a small set of accurate annotations. It provides a theoretical bound on the performance gap under mixed supervision and proposes Learnable Reference Bank (LRB) with a teacher–student EMA to refine weak cues through sample-specific prompts. The resulting LRB-WREL framework demonstrates significant gains on RRSIS-D and RIS-LAD across varying weak-data ratios, approaching or surpassing fully supervised methods, especially in low-annotation scenarios. This annotation-efficient approach offers practical benefits for scalable remote sensing segmentation and cross-modal understanding.

Abstract

Referring Remote Sensing Image Segmentation (RRSIS) aims to segment instances in remote sensing images according to referring expressions. Unlike Referring Image Segmentation on general images, acquiring high-quality referring expressions in the remote sensing domain is particularly challenging due to the prevalence of small, densely distributed objects and complex backgrounds. This paper introduces a new learning paradigm, Weakly Referring Expression Learning (WREL) for RRSIS, which leverages abundant class names as weakly referring expressions together with a small set of accurate ones to enable efficient training under limited annotation conditions. Furthermore, we provide a theoretical analysis showing that mixed-referring training yields a provable upper bound on the performance gap relative to training with fully annotated referring expressions, thereby establishing the validity of this new setting. We also propose LRB-WREL, which integrates a Learnable Reference Bank (LRB) to refine weakly referring expressions through sample-specific prompt embeddings that enrich coarse class-name inputs. Combined with a teacher-student optimization framework using dynamically scheduled EMA updates, LRB-WREL stabilizes training and enhances cross-modal generalization under noisy weakly referring supervision. Extensive experiments on our newly constructed benchmark with varying weakly referring data ratios validate both the theoretical insights and the practical effectiveness of WREL and LRB-WREL, demonstrating that they can approach or even surpass models trained with fully annotated referring expressions.

Figures (2)

• Figure 1: Challenges in obtaining high-quality referring expressions in remote sensing images and the resulting performance gains from supplementing training with class names as weakly referring expressions.
• Figure 2: Overview of the proposed LRB-WREL framework, which integrates weakly referring samples with accurate samples through a learnable reference bank and teacher–student optimization.