MMM-RS: A Multi-modal, Multi-GSD, Multi-scene Remote Sensing Dataset and Benchmark for Text-to-Image Generation

TL;DR

This paper proposes a Multi-modal, Multi-GSD, Multi-scene Remote Sensing (MMM-RS) dataset and benchmark for text-to-image generation in diverse remote sensing scenarios, and utilizes a large-scale pretrained vision-language model to automatically output text prompts and perform hand-crafted rectification, resulting in information-rich text-image pairs.

Abstract

Recently, the diffusion-based generative paradigm has achieved impressive general image generation capabilities with text prompts due to its accurate distribution modeling and stable training process. However, generating diverse remote sensing (RS) images that are tremendously different from general images in terms of scale and perspective remains a formidable challenge due to the lack of a comprehensive remote sensing image generation dataset with various modalities, ground sample distances (GSD), and scenes. In this paper, we propose a Multi-modal, Multi-GSD, Multi-scene Remote Sensing (MMM-RS) dataset and benchmark for text-to-image generation in diverse remote sensing scenarios. Specifically, we first collect nine publicly available RS datasets and conduct standardization for all samples. To bridge RS images to textual semantic information, we utilize a large-scale pretrained vision-language model to automatically output text prompts and perform hand-crafted rectification, resulting in information-rich text-image pairs (including multi-modal images). In particular, we design some methods to obtain the images with different GSD and various environments (e.g., low-light, foggy) in a single sample. With extensive manual screening and refining annotations, we ultimately obtain a MMM-RS dataset that comprises approximately 2.1 million text-image pairs. Extensive experimental results verify that our proposed MMM-RS dataset allows off-the-shelf diffusion models to generate diverse RS images across various modalities, scenes, weather conditions, and GSD. The dataset is available at https://github.com/ljl5261/MMM-RS.

Figures (8)

• Figure 1: The samples from different RS dataset. (a) shows a sample in the classic RSICD dataset RSICD, which is a simple text-image pair. (b) shows a sample in the classic SEN1-2 dataset SEN1-2, which is a multi-modal image pair including RGB image and SAR image. (c) shows a sample in our proposed MMM-RS dataset. In contrast, our MMM-RS dataset provides not only multi-modal image pair but also information-rich text prompt including simple text prompt describing image content, GSD level (i.e., spatial resolution), type of weather and satellite (different color for better observation).
• Figure 2: MMM-RS dataset statistics from different aspects.
• Figure 3: The framework of information-rich text prompt generation.
• Figure 4: The framework of multi-scene remote sensing image synthesis.
• Figure 5: An example for generating different GSD images for the same sample.