EarthMarker: A Visual Prompting Multi-modal Large Language Model for Remote Sensing

TL;DR

EarthMarker addresses the need for fine-grained interpretability in remote sensing imagery by introducing a visual prompting multi-modal LLM that operates at image, region, and point levels. It leverages a Mixture of Visual Experts to fuse multi-scale visual features with visual prompts, and employs a three-stage cross-domain training regime to transfer general knowledge from natural scenes to RS data while preserving RS-specific capabilities. The authors also build RSVP, a large RS visual-prompting dataset, to enable instruction-following across granularities, supported by GPT-4V-generated data for richer reasoning. Across scene classification, captioning, and region/relationship tasks, EarthMarker demonstrates superior zero-shot and supervised performance, highlighting the practical value for RS analysis and interactive geospatial understanding.

Abstract

Recent advances in prompt learning have allowed users to interact with artificial intelligence (AI) tools in multi-turn dialogue, enabling an interactive understanding of images. However, it is difficult and inefficient to deliver information in complicated remote sensing (RS) scenarios using plain language instructions alone, which would severely hinder deep comprehension of the latent content in imagery. Besides, existing prompting strategies in natural scenes are hard to apply to interpret the RS data due to significant domain differences. To address these challenges, the first visual prompting-based multi-modal large language model (MLLM) named EarthMarker is proposed in the RS domain. EarthMarker is capable of interpreting RS imagery at the image, region, and point levels by levering visual prompts (i.e., boxes and points). Specifically, a shared visual encoding method is developed to establish the spatial pattern interpretation relationships between the multi-scale representations of input images and various visual prompts. Subsequently, the mixed visual-spatial representations are associated with language instructions to construct joint prompts, enabling the interpretation of intricate content of RS imagery. Furthermore, to bridge the domain gap between natural and RS data, and effectively transfer domain-level knowledge from natural scenes to the RS domain, a cross-domain learning strategy is developed to facilitate the RS imagery understanding. In addition, to tackle the lack of RS visual prompting data, a dataset named RSVP featuring multi-modal multi-granularity visual prompts instruction-following is constructed. Our code and dataset are available at https://github.com/wivizhang/EarthMarker.

Figures (7)

• Figure 1: Examples of multi-granularity (e.g., image-level, region-level, and point-level) RS imagery interpretation by the proposed EarthMarker, which excels in various visual tasks including scene classification, referring object classification, captioning, relationship analyses, etc.
• Figure 2: (a) Overall model architecture of the proposed EarthMarker; (b) Cross-domain training.
• Figure 3: The referring object classification results on RS images demonstrate superior region-level RS visual understanding capability of EarthMarker compared to other MLLMs and visual prompting models (the symbol $\surd$ indicates consistency with the ground truth, the symbol ${\times}$ represents all incorrect answers, the yellow highlights denote errors, and the blue text represents relatively correct responses.).
• Figure 4: The brief region captioning results on RS images demonstrate the precise fine-grained object-level RS visual comprehension ability of EarthMarker, compared to other MLLMs, visual prompting models (the symbol $\surd$ indicates consistency with the ground truth, the symbol ${\times}$ represents all incorrect answers, the yellow highlights denote errors, and the blue text represents relatively correct responses).
• Figure 5: Examples of complex low-resolution cases of EarthMarker. EarthMarker can successfully interpret scenes that exhibit challenges such as blurriness, color distortion, and noise interference.