Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Using Game Engines and Machine Learning to Create Synthetic Satellite Imagery for a Tabletop Verification Exercise

Johannes Hoster, Sara Al-Sayed, Felix Biessmann, Alexander Glaser, Kristian Hildebrand, Igor Moric, Tuong Vy Nguyen

TL;DR

The paper tackles the scarcity of high-resolution, timely satellite imagery for nuclear-monitoring by proposing a synthetic-imagery pipeline that fuses a Unity3D game engine with multimodal ML. It renders a procedurally generated nuclear-plant scene under controllable conditions (time-of-day, cloud cover, off-nadir angle, activity) and refines the output with a CoAdapter model using canny edges, depth maps, sketches, and text prompts, achieving up to $2^4=16$ input-modality combinations. The approach enables rapid generation of large, varied datasets suitable for tabletop verification exercises and constellation-design studies, while acknowledging limitations in photorealism and the potential for bias. This work lays groundwork for evaluating long-term satellite-imaging potential in verification and governance contexts, and it calls for careful consideration of dual-use risks and ethical safeguards as synthetic imagery becomes more capable.

Abstract

Satellite imagery is regarded as a great opportunity for citizen-based monitoring of activities of interest. Relevant imagery may however not be available at sufficiently high resolution, quality, or cadence -- let alone be uniformly accessible to open-source analysts. This limits an assessment of the true long-term potential of citizen-based monitoring of nuclear activities using publicly available satellite imagery. In this article, we demonstrate how modern game engines combined with advanced machine-learning techniques can be used to generate synthetic imagery of sites of interest with the ability to choose relevant parameters upon request; these include time of day, cloud cover, season, or level of activity onsite. At the same time, resolution and off-nadir angle can be adjusted to simulate different characteristics of the satellite. While there are several possible use-cases for synthetic imagery, here we focus on its usefulness to support tabletop exercises in which simple monitoring scenarios can be examined to better understand verification capabilities enabled by new satellite constellations and very short revisit times.

Using Game Engines and Machine Learning to Create Synthetic Satellite Imagery for a Tabletop Verification Exercise

TL;DR

The paper tackles the scarcity of high-resolution, timely satellite imagery for nuclear-monitoring by proposing a synthetic-imagery pipeline that fuses a Unity3D game engine with multimodal ML. It renders a procedurally generated nuclear-plant scene under controllable conditions (time-of-day, cloud cover, off-nadir angle, activity) and refines the output with a CoAdapter model using canny edges, depth maps, sketches, and text prompts, achieving up to input-modality combinations. The approach enables rapid generation of large, varied datasets suitable for tabletop verification exercises and constellation-design studies, while acknowledging limitations in photorealism and the potential for bias. This work lays groundwork for evaluating long-term satellite-imaging potential in verification and governance contexts, and it calls for careful consideration of dual-use risks and ethical safeguards as synthetic imagery becomes more capable.

Abstract

Satellite imagery is regarded as a great opportunity for citizen-based monitoring of activities of interest. Relevant imagery may however not be available at sufficiently high resolution, quality, or cadence -- let alone be uniformly accessible to open-source analysts. This limits an assessment of the true long-term potential of citizen-based monitoring of nuclear activities using publicly available satellite imagery. In this article, we demonstrate how modern game engines combined with advanced machine-learning techniques can be used to generate synthetic imagery of sites of interest with the ability to choose relevant parameters upon request; these include time of day, cloud cover, season, or level of activity onsite. At the same time, resolution and off-nadir angle can be adjusted to simulate different characteristics of the satellite. While there are several possible use-cases for synthetic imagery, here we focus on its usefulness to support tabletop exercises in which simple monitoring scenarios can be examined to better understand verification capabilities enabled by new satellite constellations and very short revisit times.
Paper Structure (15 sections, 10 figures)

This paper contains 15 sections, 10 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Synthetic datasets supporting nuclear agreement monitoring:
  4. Text-conditioned image generation:
  5. Applications for dataset generation:
  6. Game engines for system design, testing, and validation:
  7. Method
  8. Imagery Generation Using Game Engine
  9. Multimodally-guided Satellite Imagery Synthesis
  10. Discussion
  11. Method Limitations and Desiderata
  12. Ethical and Political Considerations
  13. Large-scale data collection and analysis:
  14. Bias perpetuation:
  15. Conclusion

Figures (10)

  • Figure 1: Overview of our method from game engine render, via control input modality for structural guidance, to reinsertion of details into the synthesized image
  • Figure 2: Visualization of different components of the model from basic structure to high level of activity onsite
  • Figure 3: Visualization of different off-nadir angles
  • Figure 4: Visualization of different times-of-day
  • Figure 5: Visualization of different cloud coverage
  • ...and 5 more figures