HP-GAN: Harnessing pretrained networks for GAN improvement with FakeTwins and discriminator consistency

TL;DR

HP-GAN tackles GAN training instability and data efficiency by leveraging pretrained networks through FakeTwins self-supervised learning applied to generated images and a discriminator-consistency regularizer across CNN and ViT discriminators. It introduces fixed pretrained encoders as SSL teachers and enforces cross-network consensus using a dedicated loss, while employing a multi-discriminator architecture built on CNN+ViT backbones. Across 17 diverse datasets, HP-GAN achieves state-of-the-art Fréchet Inception Distance and improves recall and KID, particularly in data-limited settings, demonstrating data-efficient and diverse image synthesis. The approach provides a practical framework for integrating heterogeneous pretrained priors into GAN training and establishes a foundation for further gains via SSL objectives and backbone diversity.

Abstract

Generative Adversarial Networks (GANs) have made significant progress in enhancing the quality of image synthesis. Recent methods frequently leverage pretrained networks to calculate perceptual losses or utilize pretrained feature spaces. In this paper, we extend the capabilities of pretrained networks by incorporating innovative self-supervised learning techniques and enforcing consistency between discriminators during GAN training. Our proposed method, named HP-GAN, effectively exploits neural network priors through two primary strategies: FakeTwins and discriminator consistency. FakeTwins leverages pretrained networks as encoders to compute a self-supervised loss and applies this through the generated images to train the generator, thereby enabling the generation of more diverse and high quality images. Additionally, we introduce a consistency mechanism between discriminators that evaluate feature maps extracted from Convolutional Neural Network (CNN) and Vision Transformer (ViT) feature networks. Discriminator consistency promotes coherent learning among discriminators and enhances training robustness by aligning their assessments of image quality. Our extensive evaluation across seventeen datasets-including scenarios with large, small, and limited data, and covering a variety of image domains-demonstrates that HP-GAN consistently outperforms current state-of-the-art methods in terms of Fréchet Inception Distance (FID), achieving significant improvements in image diversity and quality. Code is available at: https://github.com/higun2/HP-GAN.

Figures (9)

• Figure 1: Algorithmic overview. The images are augmented and resized, then passed through pretrained CNN and ViT networks, along with their respective feature mixing blocks (CCM+CSM, dashed purple arrows). The processed multi-scale feature maps are then subjected to independent discriminators. During training, we impose consistency between discriminator outputs from different feature networks ($\mathcal{L}_{\mathcal{DC}}$). Lastly, we propose the FakeTwins loss ($\mathcal{L}_{\mathcal{FT}}$), which is applied through fake images, with the aim of maximizing the diversity of images. FakeTwins is an SSL based approach which utilizes pretrained networks as encoders. $\oplus$ indicates concatenation.
• Figure 2: The signed real logits of the discriminator $sign(D(\textbf{x}))$ according to Config D and E on the FFHQ dataset. Imgs (M) denotes the cumulative number of training images (in millions).
• Figure 3: FID and $\mathcal{L}_{\mathcal{FT}}$ according to Config D and E on the pokemon dataset. Imgs (M) denotes the cumulative number of training images (in millions).
• Figure 4: $\mathcal{L}_{\mathcal{FT}}$ in batches of images with various settings. Rows 1 and 2 in the left column are the results when single color image is repeated, and when multiple color images are in the batch. Rows 3 and 4 in the left column are when single face image is repeated, and when the batch consists of similar but different images generated with noise perturbation. The right column is the comparison according to the degree of Gaussian blurring. We compute 100 times with different random augmentation and report the average for each result.
• Figure 5: Generated images on benchmark dataset. Categories (top to bottom): FFHQ, LSUN-Bedroom and LSUN-Church.