Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EarthSynth: Generating Informative Earth Observation with Diffusion Models

Jiancheng Pan, Shiye Lei, Yuqian Fu, Jiahao Li, Yanxing Liu, Yuze Sun, Xiao He, Long Peng, Xiaomeng Huang, Bo Zhao

TL;DR

EarthSynth introduces a diffusion-based foundation model trained on a large, multi-source Earth observation dataset to enable multi-task, cross-satellite data generation. The core innovations, Counterfactual Composition (CF-Comp) and R-Filter, enhance diversity and data quality for open-vocabulary RSI tasks. Extensive experiments across scene classification, object detection, and semantic segmentation demonstrate that synthetic data from EarthSynth improves downstream performance, with ablations highlighting the value of local layout constraints and filtering. While promising, the approach is RGB-focused and computationally intensive, suggesting future work on multispectral generalization and efficiency improvements.

Abstract

Remote sensing image (RSI) interpretation typically faces challenges due to the scarcity of labeled data, which limits the performance of RSI interpretation tasks. To tackle this challenge, we propose EarthSynth, a diffusion-based generative foundation model that enables synthesizing multi-category, cross-satellite labeled Earth observation for downstream RSI interpretation tasks. To the best of our knowledge, EarthSynth is the first to explore multi-task generation for remote sensing, tackling the challenge of limited generalization in task-oriented synthesis for RSI interpretation. EarthSynth, trained on the EarthSynth-180K dataset, employs the Counterfactual Composition training strategy with a three-dimensional batch-sample selection mechanism to improve training data diversity and enhance category control. Furthermore, a rule-based method of R-Filter is proposed to filter more informative synthetic data for downstream tasks. We evaluate our EarthSynth on scene classification, object detection, and semantic segmentation in open-world scenarios. There are significant improvements in open-vocabulary understanding tasks, offering a practical solution for advancing RSI interpretation.

EarthSynth: Generating Informative Earth Observation with Diffusion Models

TL;DR

EarthSynth introduces a diffusion-based foundation model trained on a large, multi-source Earth observation dataset to enable multi-task, cross-satellite data generation. The core innovations, Counterfactual Composition (CF-Comp) and R-Filter, enhance diversity and data quality for open-vocabulary RSI tasks. Extensive experiments across scene classification, object detection, and semantic segmentation demonstrate that synthetic data from EarthSynth improves downstream performance, with ablations highlighting the value of local layout constraints and filtering. While promising, the approach is RGB-focused and computationally intensive, suggesting future work on multispectral generalization and efficiency improvements.

Abstract

Remote sensing image (RSI) interpretation typically faces challenges due to the scarcity of labeled data, which limits the performance of RSI interpretation tasks. To tackle this challenge, we propose EarthSynth, a diffusion-based generative foundation model that enables synthesizing multi-category, cross-satellite labeled Earth observation for downstream RSI interpretation tasks. To the best of our knowledge, EarthSynth is the first to explore multi-task generation for remote sensing, tackling the challenge of limited generalization in task-oriented synthesis for RSI interpretation. EarthSynth, trained on the EarthSynth-180K dataset, employs the Counterfactual Composition training strategy with a three-dimensional batch-sample selection mechanism to improve training data diversity and enhance category control. Furthermore, a rule-based method of R-Filter is proposed to filter more informative synthetic data for downstream tasks. We evaluate our EarthSynth on scene classification, object detection, and semantic segmentation in open-world scenarios. There are significant improvements in open-vocabulary understanding tasks, offering a practical solution for advancing RSI interpretation.
Paper Structure (27 sections, 13 equations, 15 figures, 11 tables, 2 algorithms)

This paper contains 27 sections, 13 equations, 15 figures, 11 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Counterfactual Composition for Controllable Diffusion Training
  5. Training Data Synthesis
  6. Experiment
  7. Comparative Results
  8. Ablation Studies
  9. Visualization Analysis
  10. Conclusion
  11. Technical Appendices and Supplementary Material
  12. More Preliminaries
  13. Copy-Paste Augmentation.
  14. EarthSynth-180K Dataset
  15. Random Cropping Strategy
  16. ...and 12 more sections

Figures (15)

  • Figure 1: A diffusion-based generative foundation model, EarthSynth, pretrained on multi-source and multi-category data, synthesizing Earth observation with a semantic mask and text for downstream remote sensing image interpretation tasks.
  • Figure 2: EarthSynth is trained with CF-Comp training strategy on real and unrealistic data distribution, learns remote sensing pixel-level properties in multiple dimensions, and builds a unified process for conditional diffusion training and synthesis.
  • Figure 3: Left: Copy-Paste used in CF-Comp Strategy. Right: CLIP-based rule filtering retains high-quality images.
  • Figure 4: Effect of samples per class on DOTAv2 dataset.
  • Figure 5: Left: Visualization of synthesis satellite images on DOTAv2 dataset. Right: EarthSynth can generate some unrealistic logical scenes controlled by different text prompts.
  • ...and 10 more figures

Theorems & Definitions (2)

  • Definition 1: Counterfactual Composition
  • Remark 1