EarthVL: A Progressive Earth Vision-Language Understanding and Generation Framework

TL;DR

EarthVLSet provides a multi-task, geography-focused dataset that connects image, mask, and text for sub-meter remote sensing scenes, enabling comprehensive evaluation of segmentation-guided VQA. EarthVLNet progressively fuses land-cover segmentation with an object-aware LLM and a novel ND loss to jointly optimize object counting and open-ended generation, achieving improved performance across semantic segmentation and both multiple-choice and open-ended VQA tasks. The work demonstrates that pixel-level semantics enhance VQA, that vision encoders drive MC-VQA gains more than language decoders, and that open-ended VQA benefits from strong vision and language models, validated by both automatic and human assessments. Practically, this framework supports city planning decisions and urban-heat mitigation strategies by providing interpretable, relational reasoning and actionable text grounded in geospatial object semantics.

Abstract

Earth vision has achieved milestones in geospatial object recognition but lacks exploration in object-relational reasoning, limiting comprehensive scene understanding. To address this, a progressive Earth vision-language understanding and generation framework is proposed, including a multi-task dataset (EarthVLSet) and a semantic-guided network (EarthVLNet). Focusing on city planning applications, EarthVLSet includes 10.9k sub-meter resolution remote sensing images, land-cover masks, and 761.5k textual pairs involving both multiple-choice and open-ended visual question answering (VQA) tasks. In an object-centric way, EarthVLNet is proposed to progressively achieve semantic segmentation, relational reasoning, and comprehensive understanding. The first stage involves land-cover segmentation to generate object semantics for VQA guidance. Guided by pixel-wise semantics, the object awareness based large language model (LLM) performs relational reasoning and knowledge summarization to generate the required answers. As for optimization, the numerical difference loss is proposed to dynamically add difference penalties, addressing the various objects' statistics. Three benchmarks, including semantic segmentation, multiple-choice, and open-ended VQA demonstrated the superiorities of EarthVLNet, yielding three future directions: 1) segmentation features consistently enhance VQA performance even in cross-dataset scenarios; 2) multiple-choice tasks show greater sensitivity to the vision encoder than to the language decoder; and 3) open-ended tasks necessitate advanced vision encoders and language decoders for an optimal performance. We believe this dataset and method will provide a beneficial benchmark that connects ''image-mask-text'', advancing geographical applications for Earth vision.

Figures (21)

• Figure 1: Comprehensive understanding of HSR remote sensing imagery. To automatically achieve "what locations have what objects" and "what relations form what scenes", we propose a benchmark dataset and method to connect the semantic segmentation and VQA tasks.
• Figure 2: The VQA methods can be divided into three categories, according to the vision feature type: (a) global fusion methods, (b) bounding box based methods, and (c) segmentation-based methods. The segmentation features provide more refined semantic boundaries at the pixel level, contributing accurate object statistics and relational reasoning for complex HSR scenes.
• Figure 3: Global distribution of the city planning-oriented EarthVLSet dataset. The different regions represent diverse object landscapes, spectra, and affordances, challenging the model transferability. The multi-choice QA pairs require relational reasoning (topologies, distances, sub-properties, etc.) for geospatial objects. The open-ended QA pairs provide detailed sentences for scene understanding from different aspects. As an intermediary, the semantic mask links the remote sensing imagery and geographical language
• Figure 4: Hierarchical structures of multiple-choice question categories in EarthVLSet.
• Figure 5: Distributions of multiple-choice questions in EarthVLSet dataset.