Remote Diffusion

TL;DR

This work investigates adapting diffusion-based image synthesis to remote sensing by fine-tuning Stable Diffusion v1.5 on the RSICD dataset, leveraging dataset captions for text conditioning. It also introduces a synthetic LULC dataset generated via a RAG/ChatGPT prompting workflow and trains a baseline ResNet-18 classifier on this data, along with fine-tuning a remote-sensing LLM (Phi-1.5). Quantitative evaluation using FID ($245.3629$) and qualitative expert feedback reveal suboptimal image quality and realism, driven by limited pretraining data and computational resources, despite promising potential for domain-specific generation. The study contributes an initial RS diffusion workflow, a public LULC synthetic dataset, and a detailed analysis of challenges and future directions for improving remote-sensing diffusion models and captioning in this domain.

Abstract

I explored adapting Stable Diffusion v1.5 for generating domain-specific satellite and aerial images in remote sensing. Recognizing the limitations of existing models like Midjourney and Stable Diffusion, trained primarily on natural RGB images and lacking context for remote sensing, I used the RSICD dataset to train a Stable Diffusion model with a loss of 0.2. I incorporated descriptive captions from the dataset for text-conditioning. Additionally, I created a synthetic dataset for a Land Use Land Classification (LULC) task, employing prompting techniques with RAG and ChatGPT and fine-tuning a specialized remote sensing LLM. However, I faced challenges with prompt quality and model performance. I trained a classification model (ResNet18) on the synthetic dataset achieving 49.48% test accuracy in TorchGeo to create a baseline. Quantitative evaluation through FID scores and qualitative feedback from domain experts assessed the realism and quality of the generated images and dataset. Despite extensive fine-tuning and dataset iterations, results indicated subpar image quality and realism, as indicated by high FID scores and domain-expert evaluation. These findings call attention to the potential of diffusion models in remote sensing while highlighting significant challenges related to insufficient pretraining data and computational resources.

Figures (10)

• Figure 1: Fine-Tuning
• Figure 2: Data Generation
• Figure 3: Downstream Evaluation
• Figure 4: Prompt: Barren landscape of a rocky desert canyon, Class: Bare Land
• Figure 5: Prompt: Vineyards and orchards in a wine-producing region, Class: Crop Land