Bridging Semantics and Geometry: A Decoupled LVLM-SAM Framework for Reasoning Segmentation in Remote Sensing

TL;DR

<3-5 sentence high-level summary> Think2Seg-RS introduces a decoupled LVLM–SAM framework where a trainable LVLM prompter generates structured geometric prompts for a frozen SAM to perform reasoning-driven segmentation in remote sensing. The approach uses mask-only GRPO optimization with a simple format+IoU reward, enabling the LVLM to learn prompting strategies without instance-level supervision. It achieves state-of-the-art results on EarthReason and shows strong zero-shot transfer to referring segmentation benchmarks, demonstrating robust semantic-level generalization and modularity. The findings also reveal that compact SAM variants suit semantic-level tasks better and highlight a pathway toward unified semantic–instance geospatial understanding via adaptive prompting and geometry-aware rewards.

Abstract

Large Vision-Language Models (LVLMs) hold great promise for advancing remote sensing (RS) analysis, yet existing reasoning segmentation frameworks couple linguistic reasoning and pixel prediction through end-to-end supervised fine-tuning, leading to weak geometric grounding and limited generalization across tasks. To address this, we developed Think2Seg-RS, a decoupled framework that trains an LVLM prompter to control a frozen Segment Anything Model (SAM) via structured geometric prompts. Through a mask-only reinforcement learning objective, the LVLM learns to translate abstract semantic reasoning into spatially grounded actions, achieving state-of-the-art performance on the EarthReason dataset. Remarkably, the learned prompting policy generalizes zero-shot to multiple referring segmentation benchmarks, exposing a distinct divide between semantic-level and instance-level grounding. We further found that compact segmenters outperform larger ones under semantic-level supervision, and that negative prompts are ineffective in heterogeneous aerial backgrounds. Together, these findings establish semantic-level reasoning segmentation as a new paradigm for geospatial understanding, opening the way toward unified, interpretable LVLM-driven Earth observation. Our code and model are available at https://github.com/Ricardo-XZ/Think2Seg-RS.

Figures (9)

• Figure 1: Illustration of the evolution of segmentation paradigms in RS imagery. (a) Input image sampled from the iSAID dataset waqas2019isaid. (b) Semantic segmentation ground truth provided by the dataset, assigning fixed class labels to every pixel. (c) Instance segmentation ground truth, distinguishing individual objects of the same class. (d) Referring expression segmentation result (mask and generated prompts,box in red and positive points in green) produced by our proposed Think2Seg-RS method, guided by explicit language queries. (e) Reasoning segmentation result by Think2Seg-RS, handling implicit, compositional queries.
• Figure 2: Architecture of Think2Seg-RS. A trainable LVLM prompter interprets the image–query pair, generates CoT reasoning, and outputs structured JSON prompts optimized via GRPO. These prompts are then executed by a frozen segmenter (SAM) to produce the final segmentation masks.
• Figure 3: Qualitative results of Think2Seg-RS on the EarthReason dataset. (a) Inputs, including the user query and corresponding remote sensing image. (b) Model outputs, comprising the LVLM's reasoning text from the <think> stage, the generated geometric prompts (bounding boxes in red and positive points in green), and the resulting segmentation mask predicted by SAM2. (c) Ground-truth (GT) mask for reference.
• Figure 4: Effect of SAM2 model scale on semantic-level segmentation. (a) Input image and (b) corresponding ground-truth (GT) mask illustrate a semantic-level annotation where the entire wastewater treatment plant is represented as one coherent polygon. Outputs from four SAM2 variants, namely (c) Tiny, (d) Small, (e) Base-plus, and (f) Large, show that larger models generate over-detailed, fragmented masks misaligned with the coarse annotation style, resulting in lower IoU scores. In contrast, smaller variants (Tiny and Small) produce smoother, spatially coherent masks that better match the semantic-level ground truth, achieving higher IoU values.
• Figure 5: Qualitative analysis of prompting combination strategies for SAM. Each pair displays the input image with LVLM generated prompts and the resulting segmentation mask (left, bounding boxes in red, positive points in green, negative points in red) and the ground-truth mask (right).