Adversarial Semantic Augmentation for Training Generative Adversarial Networks under Limited Data

TL;DR

This work introduces Adversarial Semantic Augmentation (ASA) for training GANs with limited data by operating in the semantic feature space rather than the image space. By estimating online covariance matrices of semantic features for real and generated data, ASA identifies meaningful transformation directions and augments features through a Gaussian perturbation, while an upper bound on the adversarial loss is minimized to realize implicit augmentation. The approach preserves the original data distribution (JS divergence to the generator is controlled) and integrates with a discriminative feature encoder enhanced by channel and spatial attention, plus a reconstruction term in the learning objective. Empirical results across few-shot and large-scale datasets show consistent improvements in fidelity and diversity, faster convergence, and strong qualitative gains with minimal computational overhead. ASA thus offers a principled, scalable path to data-efficient GANs suitable for SEO-friendly previews and reusable in diverse GAN architectures.

Abstract

Generative adversarial networks (GANs) have made remarkable achievements in synthesizing images in recent years. Typically, training GANs requires massive data, and the performance of GANs deteriorates significantly when training data is limited. To improve the synthesis performance of GANs in low-data regimes, existing approaches use various data augmentation techniques to enlarge the training sets. However, it is identified that these augmentation techniques may leak or even alter the data distribution. To remedy this, we propose an adversarial semantic augmentation (ASA) technique to enlarge the training data at the semantic level instead of the image level. Concretely, considering semantic features usually encode informative information of images, we estimate the covariance matrices of semantic features for both real and generated images to find meaningful transformation directions. Such directions translate original features to another semantic representation, e.g., changing the backgrounds or expressions of the human face dataset. Moreover, we derive an upper bound of the expected adversarial loss. By optimizing the upper bound, our semantic augmentation is implicitly achieved. Such design avoids redundant sampling of the augmented features and introduces negligible computation overhead, making our approach computation efficient. Extensive experiments on both few-shot and large-scale datasets demonstrate that our method consistently improve the synthesis quality under various data regimes, and further visualized and analytic results suggesting satisfactory versatility of our proposed method.

Figures (11)

• Figure 1: Comparison between conventional and semantic data augmentation. Conventional and semantic data augmentation perform images at the image and semantic level respectively, making them complementary to each other.
• Figure 2: Adversarial Semantic Augmentation for Training GANs under Limited Data. (left) Synthetic images of high fidelity and diversity produced by our model trained on the limited MetFace karras2020training dataset. These synthetic images share similar semantics but varying details such as age, gender, hairstyle, etc. (right) FID (lower is better) comparison results of our method with state-of-the-art method FastGAN liu2021towards on 100-shot datasets (with only $100$ training images).
• Figure 3: Activation from different layers of the network on the 100-shot-Obama dataset. Different distributional information is captured at different semantic level, and the semantic features goes from general to abstract as the network goes deeper.
• Figure 4: The overall framework of our proposed adversarial semantic augmentation(ASA) model. We augment the training data in the semantic space by translating the features of both real and fake images along meaningful transformation directions. Such directions are obtained by estimating the covariance matrices for both real and fake images. ASA-GAN performs semantic augmentation implicitly by optimizing an upper bound of adversarial loss, which is computation efficient and easy-to-implement.
• Figure 5: Qualitative comparison between our approach and current state-of-the-art models StyleGAN2 karras2020analyzing and FastGAN liu2021towards. The images are synthesized by the saved checkpoints with the lowest FID corresponding to each model. The red circles highlight areas with obvious artifacts and distortions. Zoom in for a better view.