Diverse Rare Sample Generation with Pretrained GANs

TL;DR

This work addresses the challenge of generating rare, diverse samples from high-resolution image datasets without retraining pretrained GANs. It combines gradient-based latent optimization with a multi-objective loss that balances rarity (via normalizing-flow density in a feature space), similarity to a reference, and diversity across multiple starts. By leveraging normalizing flows for differentiable density estimation in the VGG16-fc2 feature space and a multi-start framework, the method improves rarity and diversity across FFHQ, AFHQ, and MetFaces while maintaining realism within the real data manifold. The approach provides controllable parameters for rarity, similarity boundary, and diversity, enabling targeted exploration of low-density regions with applications in dataset augmentation, bias analysis, and edge-case generation. Limitations include occasional artifacts and dependence on the GAN’s capacity, suggesting future work on alternative generative models and more robust sample-selection strategies.

Abstract

Deep generative models are proficient in generating realistic data but struggle with producing rare samples in low density regions due to their scarcity of training datasets and the mode collapse problem. While recent methods aim to improve the fidelity of generated samples, they often reduce diversity and coverage by ignoring rare and novel samples. This study proposes a novel approach for generating diverse rare samples from high-resolution image datasets with pretrained GANs. Our method employs gradient-based optimization of latent vectors within a multi-objective framework and utilizes normalizing flows for density estimation on the feature space. This enables the generation of diverse rare images, with controllable parameters for rarity, diversity, and similarity to a reference image. We demonstrate the effectiveness of our approach both qualitatively and quantitatively across various datasets and GANs without retraining or fine-tuning the pretrained GANs.

Figures (20)

• Figure 1: Examples of rare samples generated by our method. Left: Our method produces diverse rare images for a single reference, with variations even within the same rare attribute (e.g., hats of different shapes and colors). Right: Generated rare attributes include accessories like hats, non-brown hair colors, extreme ages, and non-white races. "Pose" refers to head orientation, and "Acc." denotes accessories. Rare attributes are highlighted in bold.
• Figure 2: Schematic diagram for the objective function of our method. $\mathbf{x}^*=f(G(\mathbf{z}^*))$ and $\mathbf{x}_i=f(G(\mathbf{z}_i))$ for brevity.
• Figure 3: Examples of high- and low-density real and fake samples for Section 4.1.
• Figure 4: Examples of rare samples generated by our method for Section 4.1.
• Figure 5: Examples of diverse rare samples generated by our method for Section 4.1.