EarthGPT-X: A Spatial MLLM for Multi-level Multi-Source Remote Sensing Imagery Understanding with Visual Prompting

TL;DR

EarthGPT-X introduces a spatial MLLM designed for multi-source remote sensing imagery, addressing heterogeneity across optical, SAR, and infrared data. It combines a dual-prompting mechanism with visual prompts and text instructions, a large-scale M-RSVP dataset, and a cross-domain one-stage fusion training paradigm to enable unified, multi-level RS understanding. The model integrates a pixel-perception module for grounding and demonstrates superior performance across scene classification, referring object tasks, and region/pixel-level reasoning, while offering flexible interaction through point, box, and free-form prompts. This framework advances RS interpretation by enabling fine-grained, prompt-driven analyses in a single, trainable system with broad modality coverage and robust cross-domain generalization.

Abstract

Recent advances in natural-domain multi-modal large language models (MLLMs) have demonstrated effective spatial reasoning through visual and textual prompting. However, their direct transfer to remote sensing (RS) is hindered by heterogeneous sensing physics, diverse modalities, and unique spatial scales. Existing RS MLLMs are mainly limited to optical imagery and plain language interaction, preventing flexible and scalable real-world applications. In this article, EarthGPT-X is proposed, the first flexible spatial MLLM that unifies multi-source RS imagery comprehension and accomplishes both coarse-grained and fine-grained visual tasks under diverse visual prompts in a single framework. Distinct from prior models, EarthGPT-X introduces: 1) a dual-prompt mechanism combining text instructions with various visual prompts (i.e., point, box, and free-form) to mimic the versatility of referring in human life; 2) a comprehensive multi-source multi-level prompting dataset, the model advances beyond holistic image understanding to support hierarchical spatial reasoning, including scene-level understanding and fine-grained object attributes and relational analysis; 3) a cross-domain one-stage fusion training strategy, enabling efficient and consistent alignment across modalities and tasks. Extensive experiments demonstrate that EarthGPT-X substantially outperforms prior nature and RS MLLMs, establishing the first framework capable of multi-source, multi-task, and multi-level interpretation using visual prompting in RS scenarios.

Figures (10)

• Figure 1: Overview of the proposed framework for flexible and comprehensive spatial understanding in RS imagery: (a) Tasks: a wide range of multi-level visual reasoning tasks; (b) Model: EarthGPT-X integrates an image encoder, a visual prompt encoder, a text tokenizer, an LLM, and a pixel perception module; (c) Data: the training dataset M-RSVP is constructed from multi-source RS data (i.e., optical, synthetic aperture radar (SAR), infrared) and further enriched with GPT-4V–based annotations.
• Figure 2: Overview of the proposed EarthGPT-X architecture. It integrates images, visual prompts, and text instructions through a hybrid signal interaction mechanism, and employs a unified one-stage fusion training strategy to enable multi-source knowledge integration and multi-level multi-task reasoning.
• Figure 3: (a) Box prompts: An example of converting SAR object detection data into multi-modal conversation training data. The training data consists of three parts: the image, the visual prompt, and the text (question and answer). The image is taken directly from the object detection dataset, while the visual prompts are from the ground-truth bounding boxes. The text prompts can be selected from fixed prompt templates (*note: enhanced by GPT-4V). (b) Point prompts: each image in the semantic segmentation dataset is divided into 32 × 32 patches, randomly sampled within each patch as the point prompts, with the category retrieved from the corresponding segmentation map. (c) Free-form prompts: Simulated by applying random noise-based augmentations to regions of interest. These free-form cues emulate flexible user interactions, enabling the model to generalize beyond structured visual prompts.
• Figure 4: Illustration of the standardized prompt templates designed for GPT-4V annotation generation. The (a) part shows the public templates for generic data construction, while the (b) part demonstrates the customized task templates, $<\mathrm{Role}>$ and $<\mathrm{Format}>$, tailored to multi-source RS imagery. These templates embed object categories explicitly, ensuring the correctness and consistency of generated annotations.
• Figure 5: Image brief and detailed captioning results on optical, infrared, and SAR modalities, demonstrating the superior scene-level visual understanding ability of EarthGPT-X.