SRMF: A Data Augmentation and Multimodal Fusion Approach for Long-Tail UHR Satellite Image Segmentation

TL;DR

The paper tackles long-tail semantic segmentation in ultra-high-resolution satellite imagery by introducing SRMF, a framework that couples data augmentation with multimodal knowledge injection. The core methods are Multi-Scale Anchored Region Sampling (MSAR) for diverse region sampling, Semantic Reranking and Resampling for Training Augmentation (SRR-TA) to balance tail classes, and the Injection of General Representation Knowledge using text features from a remote-sensing Vision-Language Model. Across URUR, GID, and FBP, SRMF achieves state-of-the-art performance with gains of $3.33\%$, $0.66\%$, and $0.98\%$ in $mIoU$, respectively, demonstrating the value of cross-modal features for RS segmentation. The approach also analyzes sampling strategies and ablations, showing that text features provide robust knowledge when aligned with SAM-HQ-derived regions, though limitations persist when text coverage or tail-region detection is incomplete. Overall, SRMF advances RS segmentation by addressing long-tail distributions and highlighting the potential of multimodal fusion for practical, large-scale geospatial analysis.

Abstract

The long-tail problem presents a significant challenge to the advancement of semantic segmentation in ultra-high-resolution (UHR) satellite imagery. While previous efforts in UHR semantic segmentation have largely focused on multi-branch network architectures that emphasize multi-scale feature extraction and fusion, they have often overlooked the importance of addressing the long-tail issue. In contrast to prior UHR methods that focused on independent feature extraction, we emphasize data augmentation and multimodal feature fusion to alleviate the long-tail problem. In this paper, we introduce SRMF, a novel framework for semantic segmentation in UHR satellite imagery. Our approach addresses the long-tail class distribution by incorporating a multi-scale cropping technique alongside a data augmentation strategy based on semantic reordering and resampling. To further enhance model performance, we propose a multimodal fusion-based general representation knowledge injection method, which, for the first time, fuses text and visual features without the need for individual region text descriptions, extracting more robust features. Extensive experiments on the URUR, GID, and FBP datasets demonstrate that our method improves mIoU by 3.33\%, 0.66\%, and 0.98\%, respectively, achieving state-of-the-art performance. Code is available at: https://github.com/BinSpa/SRMF.git.

Figures (7)

• Figure 1: The distribution of class pixel counts in the training set of FBP dataset.
• Figure 2: The architectural overview of our proposed approach is presented, wherein MARS denotes Multi-Scale Anchored Region Sampling, and SRR-TA signifies Semantic Reranking and Resampling for Training Augmentation.
• Figure 3: Pre-extraction of multi-scale ground objects in three datasets.
• Figure 4: The distribution of class pixel counts in the training set of URUR, GID, WHU-OPT-SAR and DeepGlobe datasets.
• Figure 5: A comparative analysis of the SRMF method with other state-of-the-art (SOTA) approaches on the FBP dataset is presented. In (a), it can be observed that our method performs better on the two easily confused classes of urban residential and rural residential, with fewer misclassified areas. In (b), despite the presence of misclassifications, our model demonstrates a clear advantage over other models in distinguishing between arbor forest and irrigated field classes. In (c), while other models confuse railway station and industrial area, our model accurately differentiates between them. In (d), other models confuse the classes of river and pond, but our model does not require a broader context to accurately distinguish the features of river and pond, showcasing the superiority of our model in differentiating classes with very similar characteristics.