Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TSGAN: An Optical-to-SAR Dual Conditional GAN for Optical based SAR Temporal Shifting

Moien Rangzan, Sara Attarchi, Richard Gloaguen, Seyed Kazem Alavipanah

TL;DR

Optical-to-SAR translation is inherently ill-posed due to varying SAR viewing geometries. The authors propose SAR Temporal Shifting, reframing Opt2SAR as predicting SAR changes driven by optical changes across time, implemented via Temporal Shifting GAN (TSGAN), a dual-encoder conditional GAN with Siamese-like components, attention, and a change-weighted loss. They compile a bi-temporal Sentinel-1/Sentinel-2 dataset and demonstrate improved maintenance of unchanged regions and more realistic changes, aided by GLAM/SE attention and CWL1. This approach enables leveraging legacy optical datasets for SAR analyses, reduces the Fiction effect in translations, and provides reproducible tooling for extension to new regions.

Abstract

In contrast to the well-investigated field of SAR-to-Optical translation, this study explores the lesser-investigated domain of Optical-to-SAR translation, a challenging field due to the ill-posed nature of this translation. The complexity arises as a single optical data can have multiple SAR representations based on the SAR viewing geometry. We propose a novel approach, termed SAR Temporal Shifting, which inputs an optical data from the desired timestamp along with a SAR data from a different temporal point but with a consistent viewing geometry as the expected SAR data, both complemented with a change map of optical data during the intervening period. This model modifies the SAR data based on the changes observed in optical data to generate the SAR data for the desired timestamp. Our model, a dual conditional Generative Adversarial Network (GAN), named Temporal Shifting GAN (TSGAN), incorporates a siamese encoder in both the Generator and the Discriminator. To prevent the model from overfitting on the input SAR data, we employed a change weighted loss function. Our approach surpasses traditional translation methods by eliminating the GAN's fiction phenomenon, particularly in unchanged regions, resulting in higher SSIM and PSNR in these areas. Additionally, modifications to the Pix2Pix architecture and the inclusion of attention mechanisms have enhanced the model's performance on all regions of the data. This research paves the way for leveraging legacy optical datasets, the most abundant and longstanding source of Earth imagery data, extending their use to SAR domains and temporal analyses. To foster further research, we provide the code, datasets used in our study, and a framework for generating paired SAR-Optical datasets for new regions of interest. These resources are available on github.com/moienr/TemporalGAN

TSGAN: An Optical-to-SAR Dual Conditional GAN for Optical based SAR Temporal Shifting

TL;DR

Optical-to-SAR translation is inherently ill-posed due to varying SAR viewing geometries. The authors propose SAR Temporal Shifting, reframing Opt2SAR as predicting SAR changes driven by optical changes across time, implemented via Temporal Shifting GAN (TSGAN), a dual-encoder conditional GAN with Siamese-like components, attention, and a change-weighted loss. They compile a bi-temporal Sentinel-1/Sentinel-2 dataset and demonstrate improved maintenance of unchanged regions and more realistic changes, aided by GLAM/SE attention and CWL1. This approach enables leveraging legacy optical datasets for SAR analyses, reduces the Fiction effect in translations, and provides reproducible tooling for extension to new regions.

Abstract

In contrast to the well-investigated field of SAR-to-Optical translation, this study explores the lesser-investigated domain of Optical-to-SAR translation, a challenging field due to the ill-posed nature of this translation. The complexity arises as a single optical data can have multiple SAR representations based on the SAR viewing geometry. We propose a novel approach, termed SAR Temporal Shifting, which inputs an optical data from the desired timestamp along with a SAR data from a different temporal point but with a consistent viewing geometry as the expected SAR data, both complemented with a change map of optical data during the intervening period. This model modifies the SAR data based on the changes observed in optical data to generate the SAR data for the desired timestamp. Our model, a dual conditional Generative Adversarial Network (GAN), named Temporal Shifting GAN (TSGAN), incorporates a siamese encoder in both the Generator and the Discriminator. To prevent the model from overfitting on the input SAR data, we employed a change weighted loss function. Our approach surpasses traditional translation methods by eliminating the GAN's fiction phenomenon, particularly in unchanged regions, resulting in higher SSIM and PSNR in these areas. Additionally, modifications to the Pix2Pix architecture and the inclusion of attention mechanisms have enhanced the model's performance on all regions of the data. This research paves the way for leveraging legacy optical datasets, the most abundant and longstanding source of Earth imagery data, extending their use to SAR domains and temporal analyses. To foster further research, we provide the code, datasets used in our study, and a framework for generating paired SAR-Optical datasets for new regions of interest. These resources are available on github.com/moienr/TemporalGAN
Paper Structure (39 sections, 23 equations, 8 figures, 1 table)

This paper contains 39 sections, 23 equations, 8 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Generative Adversarial Networks
  4. Optical-SAR translation
  5. Translation methods
  6. Applications of optical-SAR translation
  7. Bitemporal dataset
  8. Sentinel-2 data
  9. Sentinel-1 Data
  10. Regions of interest
  11. Orbit number
  12. Despeckling
  13. Patching
  14. Methodology
  15. Base GAN architecture
  16. ...and 24 more sections

Figures (8)

  • Figure 1: Difference in the Sentinel-1 VV ascending data due to different incident angles - Bercy, Paris, France. (48.82N, 2.33E)
  • Figure 2: Flowchart of Automated GEE Paired S1S2 dataset Downloader
  • Figure 3: Architecture of the Discriminator
  • Figure 4: Architecture of the Generator
  • Figure 5: Example of the network output. $S_2$ Change Map is a false color composite where Red is the max change in RGB values, Green is the change in NIR values, and Blue is the max change in SWIR values.
  • ...and 3 more figures