EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialogues

TL;DR

EarthDial addresses the mismatch between generic vision-language models and Earth Observation data by introducing a domain-specific, multi-sensor VLM capable of handling RGB, multispectral, SAR, and temporal imagery. It employs a lightweight architecture with adaptive high-resolution tiling and a data fusion module, trained via a three-stage pipeline and an 11.11M instruction-tuning dataset to support a wide range of EO tasks in natural-language dialogue. The approach yields strong generalization across 44 downstream tasks, outperforming prior RS-VLMs and serving as a versatile tool for environmental monitoring, disaster response, and resource management. Ablation studies validate the benefits of multi-stage pretraining and multispectral fusion, confirming the design choices that enable robust temporal and spectral reasoning in remote sensing.

Abstract

Automated analysis of vast Earth observation data via interactive Vision-Language Models (VLMs) can unlock new opportunities for environmental monitoring, disaster response, and {resource management}. Existing generic VLMs do not perform well on Remote Sensing data, while the recent Geo-spatial VLMs remain restricted to a fixed resolution and few sensor modalities. In this paper, we introduce EarthDial, a conversational assistant specifically designed for Earth Observation (EO) data, transforming complex, multi-sensory Earth observations into interactive, natural language dialogues. EarthDial supports multi-spectral, multi-temporal, and multi-resolution imagery, enabling a wide range of remote sensing tasks, including classification, detection, captioning, question answering, visual reasoning, and visual grounding. To achieve this, we introduce an extensive instruction tuning dataset comprising over 11.11M instruction pairs covering RGB, Synthetic Aperture Radar (SAR), and multispectral modalities such as Near-Infrared (NIR) and infrared. Furthermore, EarthDial handles bi-temporal and multi-temporal sequence analysis for applications like change detection. Our extensive experimental results on 44 downstream datasets demonstrate that EarthDial outperforms existing generic and domain-specific models, achieving better generalization across various EO tasks. Our source codes and pre-trained models are at https://github.com/hiyamdebary/EarthDial.

Figures (8)

• Figure 1: EarthDial is the first domain-specific VLM for earth observation data that can comprehensively interpret multi-sensor imagery. Specifically, our model covers visible RGB, SAR, multi-temporal, high-res satellite and aerial imagery available in varying spatial resolutions (top half). We develop the largest remote sensing image-text instruction dataset with over 11M samples. EarthDial can perform several multimodal understanding tasks: classification, detection, captioning, visual question-answering (VQA), and grounding (bottom half). This unlocks a number of downstream applications where EarthDial shows promising results.
• Figure 2: EarthDial Architecture: The model can take a diverse set of inputs ranging from RGB to multi-spectral and time-series images. Multi-resolution inputs are converted to tokens based on an adaptive high-resolution block chen2024far that includes both local and global features. The multi-channel inputs (multi-spectral/temporal) are converted to tokens via the data fusion block, which aggregates features across all channels. The resulting visual tokens are mapped to LLM input space using MLP projectors and concatenated with the textual inputs. We use special task and modality tokens to distinguish between several input modalities and downstream tasks (Table \ref{['tab:vqa_samples']}). The LLM is trained with multimodal inputs to perform a number of downstream tasks, ranging from VQA to detection, grounding and change detection.
• Figure 3: EarthDial training Strategy for different RS modalities. We first pretrain with RGB imagery of different resolutions to achieve better alignment. Thereafter only LLM and projectors are trained on RGB and temporal inputs. We then expand the model's capability to multi-spectral and SAR imagery in Stage 3.
• Figure 4: Illustration of our versatile EarthDial model that performs across multi-modalities, multi-resolution, multispectral, and multitemporal data from diverse remote sensing applications. EarthDial extends its capabilities to a range of tasks such as scene classification, image/region-captioning, referring expression, VQA, referring expression, object detection, temporal change/disaster detection, Methane plume detection, tree species classification, UHI, and LCZs detection across multi-modalities, multi-resolution remote sensing data.
• Figure A.1: Overview of the data preparation and filtering pipeline used in the QA instruction dataset generation. The process begins with the pairing of OpenStreetMap (OSM) labels and their corresponding different sources of satellite imagery. The data goes through a label-based filtering process selecting only images with 3 labels or above, and then this data undergoes a second filtering process which is image-based to remove low-quality images. The high quality images remaining are then passed to the InternLM-XComposer2-VL model to generate question-answer pairs based on the associated reliable labels from OSM.