GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing

TL;DR

<GeoPixel> targets pixel-level grounding for high-resolution remote sensing imagery by introducing an end-to-end RS-specific large multimodal model. It combines adaptive image partitioning, a frozen vision encoder, a grounding SAM-2 encoder, and a LoRA-enabled LLM to produce precise segmentation masks alongside descriptive, geospatially grounded text. The GeoPixelD dataset provides hierarchical, multi-tier annotations (holistic, instance, cluster) linked to object masks and a controlled prompting pipeline to generate high-quality grounded conversations. Experiments on RS-GCG and referring remote sensing segmentation show clear advantages over strong RS-LMM baselines, with thorough ablations validating components such as inference resolution and annotation complexity. These advances enable fine-grained RS analysis and pave the way for robust, pixel-accurate language-grounded RS reasoning and downstream geospatial tasks.

Abstract

Recent advances in large multimodal models (LMMs) have recognized fine-grained grounding as an imperative factor of visual understanding and dialogue. However, the benefits of such representation in LMMs are limited to the natural image domain, and these models perform poorly for remote sensing (RS). The distinct overhead viewpoint, scale variation, and presence of small objects in high-resolution RS imagery present a unique challenge in region-level comprehension. Moreover, the development of the grounding conversation capability of LMMs within RS is hindered by the lack of granular, RS domain-specific grounded data. Addressing these limitations, we propose GeoPixel - the first end-to-end high resolution RS-LMM that supports pixel-level grounding. This capability allows fine-grained visual perception by generating interleaved masks in conversation. GeoPixel supports up to 4K HD resolution in any aspect ratio, ideal for high-precision RS image analysis. To support the grounded conversation generation (GCG) in RS imagery, we curate a visually grounded dataset GeoPixelD through a semi-automated pipeline that utilizes set-of-marks prompting and spatial priors tailored for RS data to methodically control the data generation process. GeoPixel demonstrates superior performance in pixel-level comprehension, surpassing existing LMMs in both single-target and multi-target segmentation tasks. Our methodological ablation studies validate the effectiveness of each component in the overall architecture. Our code and data will be publicly released.

Figures (9)

• Figure 1: An example of visually grounded detailed descriptions generated by the proposed GeoPixel, highlighting its ability to interpret and segment high-resolution remote sensing imagery with fine-grained precision. The model applies distinct masks to key objects (ground track field, swimming pool, soccer field) and semantic mask to smaller objects (vehicles). It effectively identifies spatial positions (e.g., center, top) and relationships (within the sports complex) while distinguishing between the global context (buildings, roads, green spaces) and localized structures.
• Figure 2: Overview of GeoPixel Architecture: Left: High-resolution RS images are dynamically partitioned into local patches and a resized global view, encoded by a frozen vision encoder. The encodings are projected into the language domain with separator tokens. Middle: Vision tokens, combined with text, are input into the LLM, where pLoRA is applied to vision tokens for efficient and effective multimodal alignment. Right: The corresponding embeddings for the [SEG] tokens are passed to a decoder through text projector, along with vision embeddings from the grounding vision encoders, to generate precise segmentation masks.
• Figure 3: The GeoPixelD Annotation Pipeline provides detailed multi-tier descriptions of remote sensing imagery with object phrases aligned precisely with manually annotated masks. It begins with Holistic Image Annotation (bottom left), where an LMM generates concise scene descriptions. Individual Instance Annotation (bottom right) uses spatial({pos}) and categorical ({catagorory_name}) priors with SOM ({mark_number}) prompting to describe key objects. Cluster Annotation (top right) organizes smaller or dense objects using refined grids for precise spatial analysis.
• Figure 4: Qualitative results of GeoPixel on RS-GCG. Contextually rich descriptions of RS imagery with grounded object annotations. Depending on object scale and density, it employs instance masks for precise delineation of individual objects (right and middle-right images) while semantic masks capture broader categories, such as large clusters of vehicles or small objects (middle-left and left images).
• Figure 5: Failure case due to incorrect mask association (left) and wrong instance segmentation in the same spatial region (right).