C-DiffSET: Leveraging Latent Diffusion for SAR-to-EO Image Translation with Confidence-Guided Reliable Object Generation

TL;DR

C-DiffSET tackles the challenging SAR-to-EO translation problem under limited paired data by fine-tuning a pretrained Latent Diffusion Model (LDM) and leveraging a VAE to embed SAR and EO images into a shared latent space. A novel confidence-guided diffusion loss uses per-pixel confidence maps to down-weight regions affected by temporal misalignment, improving pixel-level fidelity and reducing artifacts. The approach achieves state-of-the-art results across multiple datasets and resolutions, outperforming GAN-based and prior diffusion-based SET methods in both perceptual and structural metrics. The work introduces a scalable foundation for SAR-to-EO translation with practical implications for improved interpretation and downstream sensing tasks in remote sensing.

Abstract

Synthetic Aperture Radar (SAR) imagery provides robust environmental and temporal coverage (e.g., during clouds, seasons, day-night cycles), yet its noise and unique structural patterns pose interpretation challenges, especially for non-experts. SAR-to-EO (Electro-Optical) image translation (SET) has emerged to make SAR images more perceptually interpretable. However, traditional approaches trained from scratch on limited SAR-EO datasets are prone to overfitting. To address these challenges, we introduce Confidence Diffusion for SAR-to-EO Translation, called C-DiffSET, a framework leveraging pretrained Latent Diffusion Model (LDM) extensively trained on natural images, thus enabling effective adaptation to the EO domain. Remarkably, we find that the pretrained VAE encoder aligns SAR and EO images in the same latent space, even with varying noise levels in SAR inputs. To further improve pixel-wise fidelity for SET, we propose a confidence-guided diffusion (C-Diff) loss that mitigates artifacts from temporal discrepancies, such as appearing or disappearing objects, thereby enhancing structural accuracy. C-DiffSET achieves state-of-the-art (SOTA) results on multiple datasets, significantly outperforming the very recent image-to-image translation methods and SET methods with large margins.

Figures (20)

• Figure 1: Qualitative comparison of SAR-to-EO image translation (SET) results for very recent methods and our proposed C-DiffSET. The first row shows the SET results for full-polarization (HH, HV, VH, and VV) SAR input (ground sample distance (GSD) = 0.5$m$) of SpaceNet6 dataset, while the second and third rows exhibit the SET results for single-polarization (HH or VV) SAR input (GSD = 1$m$) of SAR2Opt and QXS-SAROPT datasets, respectively. As shown, our proposed C-DiffSET achieves superior structural accuracy and visual fidelity compared to very recent methods. Note that (i) the SET results for SpaceNet6 show better visual qualities than those of SAR2Opt and QXS-SAROPT because of using the full-polarization SAR input of SpaceNet6, and (ii) the SET results for SAR2Opt are of relatively higher visual qualities than those for QXS-SAROPT due to their less noisy SAR samples of SAR2Opt.
• Figure 2: Examples of misalignments and discrepancies in paired SAR-EO datasets. Left: Local spatial misalignments caused by sensor differences or acquisition conditions. Right: Temporal discrepancies where objects (e.g., ships) appear or disappear between SAR and EO images due to their different acquisition times.
• Figure 3: Overall framework of our Confidence Diffusion for SAR-to-EO Translation (C-DiffSET).
• Figure 4: Results of applying the VAE encoder and decoder from LDM to EO and SAR images. The first row shows input images, including EO and SAR images with different levels of speckle noise. The second row presents the corresponding VAE reconstructions, illustrating that both EO and SAR images are accurately reconstructed despite noise variations.
• Figure 5: Visual comparison of SET results on the SpaceNet6 and SAR2Opt datasets. 1st rows: GAN-based (Pix2pix, CycleGAN, CFCA-SET, and StegoGAN) methods. 2nd rows: LDM-based (BBDM, ControlNet, Uni-ControlNet, DGDM, cBBDM, and C-DiffSET) methods.