Reconciling Semantic Controllability and Diversity for Remote Sensing Image Synthesis with Hybrid Semantic Embedding

TL;DR

A hybrid semantic embedding guided generative adversarial network (HySEGGAN) is presented for controllable and efficient remote sensing image synthesis that leverages hierarchical information from a single source and strikes an excellent balance between semantic controllability and diversity.

Abstract

Significant advancements have been made in semantic image synthesis in remote sensing. However, existing methods still face formidable challenges in balancing semantic controllability and diversity. In this paper, we present a Hybrid Semantic Embedding Guided Generative Adversarial Network (HySEGGAN) for controllable and efficient remote sensing image synthesis. Specifically, HySEGGAN leverages hierarchical information from a single source. Motivated by feature description, we propose a hybrid semantic Embedding method, that coordinates fine-grained local semantic layouts to characterize the geometric structure of remote sensing objects without extra information. Besides, a Semantic Refinement Network (SRN) is introduced, incorporating a novel loss function to ensure fine-grained semantic feedback. The proposed approach mitigates semantic confusion and prevents geometric pattern collapse. Experimental results indicate that the method strikes an excellent balance between semantic controllability and diversity. Furthermore, HySEGGAN significantly improves the quality of synthesized images and achieves state-of-the-art performance as a data augmentation technique across multiple datasets for downstream tasks.

Figures (8)

• Figure 1: Schematic diagram of the proposed method. (a) Traditional semantic image synthesis method park2019spade, which exhibits poor performance. (b) Introducing additional information for fused semantic embedding. For instance, tan2023inade enhances synthesized semantic features with sketches, yet it still encounters issues like class confusion in the results. (c) Proposed hybrid semantic Embedding and fine-grained semantic feedback mechanism. Our approach ensures semantic consistency and extensibility, minimizing the sacrifice of diversity. The results demonstrate that our strategy effectively enhances the quality of fine-grained synthesized images for remote sensing image synthesis tasks.
• Figure 2: The training pipeline of the hybrid semantic embedding guided GAN is illustrated as follows. The pink section represents the computation of the Geometric-informed Spatial Descriptor (GSD), the blue section denotes the HSGNet, which primarily includes the HSFMResBlock and upsampling, and the yellow section depicts the semantic refinement network. SRN is based on an encoder-decoder architecture and incorporates a fine-grained semantic feedback process. GSDs are derived from semantic masks, and both are embedded into the input generator using hybrid semantics. The loss functions $\mathcal{L}_{ref}^I$, $\mathcal{L}_{ref}^z$ and $\mathcal{L}_{ref}^S$ proposed for SRN are specifically designed yet trained jointly to enforce semantic consistency constraints and provide fine-grained feedback.
• Figure 3: Detailed structure of the HSFMResBlock used in Fig. \ref{['fig-pipeline']}. It learns pixel-level fine-grained modulation parameters from the hybrid semantic embedding and guides the modulation of normalized activations.
• Figure 4: Illustrate the architecture of encoder layers and decoder layers in semantic refinement networks as shown in Fig. \ref{['fig-SRN']}(a) and Fig. \ref{['fig-SRN']}(b), respectively.
• Figure 5: Qualitative visualization of synthesis results for various methods on the GID-15 task. The left two columns are the semantic mask and the corresponding ground truth. The other seven columns are the synthesized results. This illustration demonstrates the superiority of our approach in terms of semantic controllability and fine-grained synthesis.