Learning to Aggregate Multi-Scale Context for Instance Segmentation in Remote Sensing Images

TL;DR

This work tackles the challenge of instance segmentation in remote sensing imagery, where scale variations, arbitrary orientation, and cluttered backgrounds hinder discriminative feature learning. It introduces CATNet, a lightweight, plug-and-play framework that aggregates global context across feature (DenseFPN), spatial (SCP), and instance (HRoIE) domains to produce more discriminative per-object representations. Across iSAID, DIOR, NWPU VHR-10, and HRSID, CATNet achieves state-of-the-art or competitive results with modest computational overhead, and ablations validate the contribution of each module and their recommended DenseFPN→SCP→HRoIE order. The approach offers practical benefits for RS perception tasks and suggests future integration with transformer-based architectures for even more robust context modeling.

Abstract

The task of instance segmentation in remote sensing images, aiming at performing per-pixel labeling of objects at instance level, is of great importance for various civil applications. Despite previous successes, most existing instance segmentation methods designed for natural images encounter sharp performance degradations when they are directly applied to top-view remote sensing images. Through careful analysis, we observe that the challenges mainly come from the lack of discriminative object features due to severe scale variations, low contrasts, and clustered distributions. In order to address these problems, a novel context aggregation network (CATNet) is proposed to improve the feature extraction process. The proposed model exploits three lightweight plug-and-play modules, namely dense feature pyramid network (DenseFPN), spatial context pyramid (SCP), and hierarchical region of interest extractor (HRoIE), to aggregate global visual context at feature, spatial, and instance domains, respectively. DenseFPN is a multi-scale feature propagation module that establishes more flexible information flows by adopting inter-level residual connections, cross-level dense connections, and feature re-weighting strategy. Leveraging the attention mechanism, SCP further augments the features by aggregating global spatial context into local regions. For each instance, HRoIE adaptively generates RoI features for different downstream tasks. Extensive evaluations of the proposed scheme on iSAID, DIOR, NWPU VHR-10, and HRSID datasets demonstrate that the proposed approach outperforms state-of-the-arts under similar computational costs. Source code and pre-trained models are available at https://github.com/yeliudev/CATNet.

Figures (11)

• Figure 1: Illustration of global context aggregation. Processed by the proposed modules, object features are enhanced by aggregating global visual context, as can be seen from the more discriminative feature maps.
• Figure 2: The challenges of performing instance segmentation in remote sensing images. Here, scale variation, arbitrary orientation, and clustered distribution lead to complicated object patterns while low contrast and cluttered background bring interfering information from the background.
• Figure 3: Overall architecture of the proposed framework. The process of global context aggregation is realized by three modules, namely a) dense feature pyramid network, b) spatial context pyramid, and c) hierarchical region of interest extractor. These modules are designed to aggregate global context information from different feature pyramid levels, spatial positions, and receptive fields at feature, spatial, and instance domains, respectively.
• Figure 4: Detailed procedure of multi-scale feature propagation in dense feature pyramid network. $\oplus$ and $\ooalign{$$\bigcirc$$\cr$$-$$\cr$$-$$}$ denote element-wise addition and ReLU activation, respectively. The cross-level feature maps are adaptively combined using a feature re-weighting strategy.
• Figure 5: Designs of spatial context modules. Permutations of feature maps are represented as their dimensions, e.g.,$C \! \times \! H \! \times \! W$ indicates a matrix with number of channels $C$, height $H$, and width $W$. $\otimes$, $\odot$, and $\oplus$ denote batched matrix multiplication, broadcast hadamard product, and broadcast element-wise addition, respectively. Convolution layers used for attention map generation, feature mapping, and context refinement are annotated as blue, red, and purple.