RSTeller: Scaling Up Visual Language Modeling in Remote Sensing with Rich Linguistic Semantics from Openly Available Data and Large Language Models

TL;DR

This work tackles the data bottleneck in remote sensing vision-language modeling by introducing an automated workflow that leverages openly available data (Google Earth Engine NAIP imagery and OpenStreetMap) and large language models to generate semantically rich caption pairs at scale. The authors present RSTeller, a dataset of over 1.3 million RS image patches, each with two captions (about 2.6 million image-text pairs), with captions that are richer than prior RS datasets as measured by diversity metrics. They validate the dataset by continual pre-training CLIP-based VLMs, showing improvements in zero-shot classification and retrieval on RS benchmarks, and perform ablations to quantify the contributions of data components and domain-scale data. The workflow reduces manual annotation effort and democratizes access to high-quality RS data, offering a scalable path for advancing RS visual-language understanding while enabling broader participation in RS research and applications.

Abstract

Abundant, well-annotated multimodal data in remote sensing are pivotal for aligning complex visual remote sensing (RS) scenes with human language, enabling the development of specialized vision language models across diverse RS interpretation tasks. However, annotating RS images with rich linguistic semantics at scale demands expertise in RS and substantial human labor, making it costly and often impractical. In this study, we propose a workflow that leverages large language models (LLMs) to generate multimodal datasets with semantically rich captions at scale from plain OpenStreetMap (OSM) data for images sourced from the Google Earth Engine (GEE) platform. This approach facilitates the generation of paired remote sensing data and can be readily scaled up using openly available data. Within this framework, we present RSTeller, a multimodal dataset comprising over 1.3 million RS images, each accompanied by two descriptive captions. Extensive experiments demonstrate that RSTeller enhances the performance of multiple existing vision language models for RS scene understanding through continual pre-training. Our methodology significantly reduces the manual effort and expertise needed for annotating remote sensing imagery while democratizing access to high-quality annotated data. This advancement fosters progress in visual language modeling and encourages broader participation in remote sensing research and applications. The RSTeller dataset is available at https://github.com/SlytherinGe/RSTeller.

Figures (12)

• Figure 1: Comparison of existing image-text multimodal RS datasets. The Measure of Textual Lexical Diversity (MTLD) scores quantify the semantic richness of the captions. All captions were tokenized using the Natural Language Toolkit (NLTK) word tokenizer, and the color of each point indicates the scale of the tokens. The proposed RSTeller dataset is marked with a star. Its MTLD score is more than twice that of the next most diverse dataset, while also being substantial in data size.
• Figure 2: Workflow for dataset generation. The left section outlines the main processes: data fetching, LLM-based captioning, and final dataset compilation. The right section details the specific captioning tasks that enhance the quality and richness of the generated captions.
• Figure 3: Inspection of the RSTeller dataset, showcasing four image patches with corresponding OSM tags, captions, and their revisions. The map highlights sample locations, while the figure illustrates OSM area and non-area elements with masked boundaries or lines. Crossed-out tags indicate filtered elements excluded from captioning. Captions represent results from Task 1 or Task 2, while revisions are outcomes of Task 3.
• Figure 4: A brief showcase of the prompt template for Task 1. Details are replaced by ellipses for brevity. Placeholders are wrapped in braces and are to be filled dynamically by the task.
• Figure 5: A brief showcase of the prompt template for Task 3. Details are replaced by ellipses for brevity. Placeholders are wrapped in braces and are to be filled dynamically by the task.