Manifold Constraint Regularization for Remote Sensing Image Generation

TL;DR

This paper analyzes the characteristics of remote sensing images and proposes manifold constraint regularization, a novel approach that tackles overfitting of GANs on remote sensing images for the first time and enhances the quality of the generated images.

Abstract

Generative Adversarial Networks (GANs) have shown notable accomplishments in remote sensing domain. However, this paper reveals that their performance on remote sensing images falls short when compared to their impressive results with natural images. This study identifies a previously overlooked issue: GANs exhibit a heightened susceptibility to overfitting on remote sensing images.To address this challenge, this paper analyzes the characteristics of remote sensing images and proposes manifold constraint regularization, a novel approach that tackles overfitting of GANs on remote sensing images for the first time. Our method includes a new measure for evaluating the structure of the data manifold. Leveraging this measure, we propose the manifold constraint regularization term, which not only alleviates the overfitting problem, but also promotes alignment between the generated and real data manifolds, leading to enhanced quality in the generated images. The effectiveness and versatility of this method have been corroborated through extensive validation on various remote sensing datasets and GAN models. The proposed method not only enhances the quality of the generated images, reflected in a 3.13\% improvement in Frechet Inception Distance (FID) score, but also boosts the performance of the GANs on downstream tasks, evidenced by a 3.76\% increase in classification accuracy.

Figures (6)

• Figure 1: The horizontal axis indicates the training process (the number of real images shown to the discriminator). (a) Training curves of StyleGAN2 on NWPU and FFHQ datasets. We randomly sample 30,000 training images from these two datasets, respectively. The GAN model diverges earlier when trained on NWPU dataset. (b) The outputs of the discriminator during training on NWPU dataset. As training progresses, the validation set of real images is misclassified as generated images, highlighting the discriminator’s overfitting issue on NWPU dataset.
• Figure 2: (a) Training curves of BigGAN on TinyImageNet, FFHQ, NWPU and PN datasets. The horizontal axis is the number of training steps. (b) Training curves of StyleGAN2 on TinyImageNet, FFHQ, NWPU and PN datasets. The horizontal axis indicates the training process (the number of real images shown to the discriminator). These training curves highlight an earlier divergence on RS datasets compared to natural ones, emphasizing our claim about the extent of overfitting in RS data.
• Figure 3: The overall architecture of our method. $G$ and $D$ are generator and discriminator. $F$ is the MLP network for relationship matrix $C$. $\mathcal{M}$ represents manifold, and $\mathcal{S}$ represents subspace.
• Figure 4: The horizontal axis indicates the training process (the number of real images shown to the discriminator). (a) Training curves of our method on NWPU dataset. (b) The outputs of the discriminator during training on NWPU dataset. (c) Training curves of our method on PN dataset. (d) The outputs of the discriminator during training on PN dataset. No divergence occurs during training, and the discriminator maintains high accuracy on the validation set. These findings suggest that MCR effectively alleviates the discriminator’s overfitting issue.
• Figure 5: (a) Training curves of BigGAN. (b) Training curves of StyleGAN2. Our method (MCR) not only reduces discriminator overfitting but also attains superior quality scores.

Theorems & Definitions (3)

• Proposition 1
• Proof 1
• Theorem 1