Generating Physically-Consistent Satellite Imagery for Climate Visualizations

TL;DR

A generative adversarial network is trained to create synthetic satellite imagery of future flooding and reforestation events and it is found that a pure deep learning-based model can generate photorealistic flood visualizations but hallucinate floods at locations that are not susceptible to flooding.

Abstract

Deep generative vision models are now able to synthesize realistic-looking satellite imagery. But, the possibility of hallucinations prevents their adoption for risk-sensitive applications, such as generating materials for communicating climate change. To demonstrate this issue, we train a generative adversarial network (pix2pixHD) to create synthetic satellite imagery of future flooding and reforestation events. We find that a pure deep learning-based model can generate photorealistic flood visualizations but hallucinates floods at locations that were not susceptible to flooding. To address this issue, we propose to condition and evaluate generative vision models on segmentation maps of physics-based flood models. We show that our physics-conditioned model outperforms the pure deep learning-based model and a handcrafted baseline. We evaluate the generalization capability of our method to different remote sensing data and different climate-related events (reforestation). We publish our code and dataset which includes the data for a third case study of melting Arctic sea ice and $>$30,000 labeled HD image triplets -- or the equivalent of 5.5 million images at 128x128 pixels -- for segmentation guided image-to-image translation in Earth observation. Code and data is available at \url{https://github.com/blutjens/eie-earth-public}.

Figures (14)

• Figure 1: We synthesize satellite imagery that visualizes flooding (right). We designed the underlying generative vision model to project flooding only in locations that are consistent with a physics-based flood model. The new visualizations could facilitate intuitive and trustworthy communication of climate risks, for example, via tabletop exercises as seen on the left. Explore more results at https://climate-viz.github.io/.
• Figure 2: Top: Model Architecture. Our model leverages the semantic image synthesis model, Pix2pixHD wang18pix2pixhd, and combines a pre-flood satellite image with a physics-based flood map to generate post-flood satellite imagery. Bottom: Arctic sea ice melt. We publish a dataset of 19446 labeled image triplets for segmentation guided image-to-image translation which includes an additional case study on melting Arctic sea ice.
• Figure 3: Our physically-consistent satellite imagery (c) could enable more engaging and relatable communication of city-scale flood risks Sheppard_2012. Most existing visualizations of coastal floods or sea-level rise that are aimed towards the public rely on color-coded geospatial rasters (a), that can be unrelatable or impersonal NoaaSlosh_20FloodFactor_2020ClimateCentral_18. Alternative photorealistic visualizations are often limited to local street-level imagery (b) Strauss_2015Schmidt_2019 and lack further spatial context. Images, left-to-right: NoaaSlosh_20NoaaSlosh_20Strauss_2015Strauss_2015Gupta_2019, ours.
• Figure 4: Our model, titled the Earth Intelligence Engine, visualizes how flooding (left) or reforestation (right) would impact the landscape as seen from space.
• Figure 5: Top: Flooding. The proposed physics-informed GAN (ours, based on wang18pix2pixhd) generates post-flood images, (d), from a pre-flood image and flood mask, (a,b). The model outperforms others, (e,f,g), in either physical-consistency or photorealism. The baseline GAN ablates the flood mask and synthesizes a fully-flooded image, (e), rendering the model physically-inconsistent. A VAE-based model Zhu_2017, creates glitchy imagery, (f), (zoom in). A handcrafted baseline model as used in common visualization tools ClimateCentral_18NOAA_2020, visualizes the correct flood extent, but is pixelated and lacks photorealism, (g). Bottom: Reforestation. The proposed reforestation mask + pix2pixHD wang18pix2pixhd GAN, (d), generates photorealistic reforestation imagery from the inputs, (a,b), outperforming handcrafted baseline models (e,f).