Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Controlling the Fidelity and Diversity of Deep Generative Models via Pseudo Density

Shuangqi Li, Chen Liu, Tong Zhang, Hieu Le, Sabine Süsstrunk, Mathieu Salzmann

TL;DR

This work introduces an approach to bias deep generative models, such as GANs and diffusion models, towards generating data with either enhanced fidelity or increased diversity through a novel metric for individual samples, named pseudo density, which is based on the nearest-neighbor information from real samples.

Abstract

We introduce an approach to bias deep generative models, such as GANs and diffusion models, towards generating data with either enhanced fidelity or increased diversity. Our approach involves manipulating the distribution of training and generated data through a novel metric for individual samples, named pseudo density, which is based on the nearest-neighbor information from real samples. Our approach offers three distinct techniques to adjust the fidelity and diversity of deep generative models: 1) Per-sample perturbation, enabling precise adjustments for individual samples towards either more common or more unique characteristics; 2) Importance sampling during model inference to enhance either fidelity or diversity in the generated data; 3) Fine-tuning with importance sampling, which guides the generative model to learn an adjusted distribution, thus controlling fidelity and diversity. Furthermore, our fine-tuning method demonstrates the ability to improve the Frechet Inception Distance (FID) for pre-trained generative models with minimal iterations.

Controlling the Fidelity and Diversity of Deep Generative Models via Pseudo Density

TL;DR

This work introduces an approach to bias deep generative models, such as GANs and diffusion models, towards generating data with either enhanced fidelity or increased diversity through a novel metric for individual samples, named pseudo density, which is based on the nearest-neighbor information from real samples.

Abstract

We introduce an approach to bias deep generative models, such as GANs and diffusion models, towards generating data with either enhanced fidelity or increased diversity. Our approach involves manipulating the distribution of training and generated data through a novel metric for individual samples, named pseudo density, which is based on the nearest-neighbor information from real samples. Our approach offers three distinct techniques to adjust the fidelity and diversity of deep generative models: 1) Per-sample perturbation, enabling precise adjustments for individual samples towards either more common or more unique characteristics; 2) Importance sampling during model inference to enhance either fidelity or diversity in the generated data; 3) Fine-tuning with importance sampling, which guides the generative model to learn an adjusted distribution, thus controlling fidelity and diversity. Furthermore, our fine-tuning method demonstrates the ability to improve the Frechet Inception Distance (FID) for pre-trained generative models with minimal iterations.
Paper Structure (16 sections, 20 figures, 3 tables, 3 algorithms)

This paper contains 16 sections, 20 figures, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Pseudo Density
  4. Density-Based Perturbation
  5. Density-Based Importance Sampling
  6. Inference with Importance Sampling
  7. Fine-tuning with Importance Sampling
  8. Experiments
  9. Conclusion
  10. Appendix
  11. Training Details, Evaluation Details, and Hyper-parameters
  12. Discriminator Output versus Pseudo Density
  13. Extra Computational Overhead
  14. Combination of Fine-tuning and Inference-Time Methods
  15. Autoregressive Generative Models
  16. ...and 1 more sections

Figures (20)

  • Figure 1: Overview of the proposed pseudo density metric. (a) The pseudo-density of each sample is inversely proportional to its average distance to its K-Nearest Neighbors in a feature space. (b) We use these pseudo density to train a density prediction network, which then can be used on any generated samples (c).
  • Figure 2: Images generated by StyleGAN2 (Top), ProjectedGAN (Middle), and IDDPM (Bottom). The training datasets are FFHQ, LSUN-Church, and LSUN-Bedroom, respectively. For each row, the left four images obtained the lowest pseudo density out of 1000 generations, and the right four obtained the highest.
  • Figure 3: Per-sample perturbation, applied to pre-trained models. Given random latent vectors $z$ and their generated images (Middle in each panel), we perturb $z$ to achieve higher pseudo density (Top in each panel) and lower pseudo density (Bottom in each panel). Groups from left to right, and from top to bottom: StyleGAN2 on FFHQ, ProjectedGAN on LSUN-Bedroom; StyleGAN3-T on FFHQ, IDDPM on LSUN-Bedroom; ProjectedGAN on LSUN-Church, ADM on LSUN-Bedroom.
  • Figure 4: Precision-recall trade-off. For all methods, the precision and recall metrics compete with each other. For polarity sampling, we used the values reported in Humayun_2022_polarity due to replication difficulty. For our method, we adopted different density thresholds $\tau$ for each dashed line with varying importance weights $w$ ranging from $0.01$ to $100$. For better visualization, we only report part of the results for polarity sampling and truncation. We visualize the Pareto frontier for polarity sampling and ours, illustrating the optimal trade-offs achieved across various hyperparameter configurations.
  • Figure 5: Images generated by pre-trained and fine-tuned models. For each column in a panel, the same latent vector is used by the pre-trained model and the fine-tuned versions for image generation. Fine-tuning with $w > 1$ biases the generative model to output more high-density data. Groups from left to right, and from top to bottom: StyleGAN2 on FFHQ, ProjectedGAN on LSUN-Bedroom; StyleGAN3-T on FFHQ, IDDPM on LSUN-Bedroom; ProjectedGAN on LSUN-Church, ADM on LSUN-Bedroom.
  • ...and 15 more figures