Dynamically Masked Discriminator for Generative Adversarial Networks

TL;DR

The paper addresses the instability in GAN training caused by the discriminator lagging behind the evolving distribution of generated data. It frames discriminator learning as online continual learning and introduces Dynamically Masked Discriminator (DMD), which detects when the discriminator slows down and dynamically masks its features to force fast adaptation. Through retardation detection and adaptive masking, DMD yields state-of-the-art FID improvements across StyleGAN-V2/StyleGAN-V3/BigGAN benchmarks and on large-scale datasets, demonstrating the practical benefit of online adaptation for GANs. The approach is plug-and-play and highlights the importance of accommodating non-stationary data distributions in guiding generator training, with broader implications for robust generative modeling.

Abstract

Training Generative Adversarial Networks (GANs) remains a challenging problem. The discriminator trains the generator by learning the distribution of real/generated data. However, the distribution of generated data changes throughout the training process, which is difficult for the discriminator to learn. In this paper, we propose a novel method for GANs from the viewpoint of online continual learning. We observe that the discriminator model, trained on historically generated data, often slows down its adaptation to the changes in the new arrival generated data, which accordingly decreases the quality of generated results. By treating the generated data in training as a stream, we propose to detect whether the discriminator slows down the learning of new knowledge in generated data. Therefore, we can explicitly enforce the discriminator to learn new knowledge fast. Particularly, we propose a new discriminator, which automatically detects its retardation and then dynamically masks its features, such that the discriminator can adaptively learn the temporally-vary distribution of generated data. Experimental results show our method outperforms the state-of-the-art approaches.

Figures (9)

• Figure 1: Illustration of the advantage of our method, where the first and second rows show generated images and feature maps taken from a discriminator layer, respectively. (a) StyleGAN-V2 introduces similar artifacts into generated images, despite the increased training steps. Our method is simple yet effectively improves the training of GANs, boosting the quality of generated samples for StyleGAN-V2 in (b). By fast learning incoming generated samples, the proposed discriminator captures artifacts in the local regions, better guiding the training of the generator. The averaging cosine similarity between the current and previous feature maps is 0.8353 for StyleGAN-V2 and 0.4232 for ours, which indicates that our method enforces the attentive regions of the discriminator to be more different than the baseline, better adapting to time-varying distribution of generated data.
• Figure 2: Illustration of time-varying distributions of generated samples, where we trace the training process of StyleGAN-V2 karras2020analyzing on FFHQ karras2019style. The mean and variance of generated 5k samples' features are computed per time interval, showing the generated distributions are dynamic and time-varying during training, as the generator evolves. kimg refers to the number of images (measured in thousand) trained so far.
• Figure 3: We trace the training of StyleGAN-V2's discriminator. The curve in (a) shows the model parameter difference of the discriminator between $t_i$ and $t_{i-1}$ in the training progress. In (b), the attentive regions of the discriminator are almost fixed at training steps when the discriminator slows down its model parameter updating, where each feature map is represented by the feature space of the discriminator trained at a time step.
• Figure 4: The pipeline of the proposed method. Our method, named Dynamically Masked Discriminator (DMD), automatically adjusts the discriminator via dynamically marking the discriminator. When DMD detects that the discriminator slows down learning, DMD dynamically assigns masks or removes masks to features of the discriminator per time interval, forcing the discriminator to learn new knowledge and preventing it from relying on old knowledge from historical samples.
• Figure 5: Illustration of the generated images, feature maps, and the corresponding masks. The feature maps are extracted from the discriminator of StyleGAN-V2 karras2020analyzing trained on FFHQ karras2019style, and the red region denotes the artifacts. By masking feature maps of the discrimination, it is possible to remove discriminative local regions/features (e.g., unrealistic regions) that are important for the sample’s distribution, shirting the sample to a new distribution.