SegEarth-R2: Towards Comprehensive Language-guided Segmentation for Remote Sensing Images

TL;DR

This work tackles the challenge of language-guided segmentation in remote sensing by introducing LaSeRS, a large-scale benchmark that captures hierarchical granularity, target multiplicity, reasoning requirements, and linguistic variability. It then presents SegEarth-R2, an efficient MLLM-based framework featuring spatial attention supervision and a flexible segmentation query mechanism to handle both single- and multi-target instructions. Empirical results show SegEarth-R2 achieves state-of-the-art performance on LaSeRS and strong cross-dataset generalization on RS referring segmentation and reasoning benchmarks, with notable gains in fine-grained, part-level segmentation at moderate model sizes. The dataset and model together set a new baseline for comprehensive, pixel-level language grounding in RS and encourage a shift toward multi-target, multi-granularity language-guided segmentation in geospatial analysis.

Abstract

Effectively grounding complex language to pixels in remote sensing (RS) images is a critical challenge for applications like disaster response and environmental monitoring. Current models can parse simple, single-target commands but fail when presented with complex geospatial scenarios, e.g., segmenting objects at various granularities, executing multi-target instructions, and interpreting implicit user intent. To drive progress against these failures, we present LaSeRS, the first large-scale dataset built for comprehensive training and evaluation across four critical dimensions of language-guided segmentation: hierarchical granularity, target multiplicity, reasoning requirements, and linguistic variability. By capturing these dimensions, LaSeRS moves beyond simple commands, providing a benchmark for complex geospatial reasoning. This addresses a critical gap: existing datasets oversimplify, leading to sensitivity-prone real-world models. We also propose SegEarth-R2, an MLLM architecture designed for comprehensive language-guided segmentation in RS, which directly confronts these challenges. The model's effectiveness stems from two key improvements: (1) a spatial attention supervision mechanism specifically handles the localization of small objects and their components, and (2) a flexible and efficient segmentation query mechanism that handles both single-target and multi-target scenarios. Experimental results demonstrate that our SegEarth-R2 achieves outstanding performance on LaSeRS and other benchmarks, establishing a powerful baseline for the next generation of geospatial segmentation. All data and code will be released at https://github.com/earth-insights/SegEarth-R2.

Figures (22)

• Figure 1: SegEarth-R2 can handle various language-guided segmentation in remote sensing across four key dimensions: (1) Hierarchical Granularity; (2) Target Multiplicity; (3) Reasoning Requirements; and (4) Linguistic Variability.
• Figure 2: Overview of the construction of LaSeRS. Our scalable, semi-automatic annotation pipeline comprises four stages: two dedicated to data generation and two dedicated to filtering. Data quality is rigorously ensured through both automatic and manual verification.
• Figure 3: Data Statistics of LaSeRS.
• Figure 4: The overall architecture of SegEarth-R2. (a) is an MLLM for interpreting the image and instruction, (b) is the segmentation head, which includes a hierarchical visual encoder for generating multi-scale image features, and two decoders for producing masks (mask decoders). (c) illustrates the formation process of the spatial attention supervision. (d) presents a visualization of the attention map from the 6th head of the 16th layer in the MLLM, with additional visual results in the Suppl. \ref{['sec:spatial_attention']}
• Figure 5: The left figure shows that as the number of segmentation queries decreases, both computational cost (TFLOPs) and inference time are correspondingly reduced. The right figure illustrates that this reduction in queries leads to a gradual increase in gIoU scores on the RRSIS-D validation and test sets.