Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Noise Dimension of GAN: An Image Compression Perspective

Ziran Zhu, Tongda Xu, Ling Li, Yan Wang

TL;DR

This work reframes GANs as discrete samplers and establishes a direct link between the required noise dimension and the bitrate needed to losslessly encode the source data. It introduces a divergence-entropy trade-off to characterize GAN performance under limited noise and provides a numerical approach (via disciplined convex-concave programming) for known source distributions. Theoretical results bound the minimum noise dimension by $n \ge \dfrac{H(X)}{26.55}$ per dimension (with refinements for bijective mappings) and connect the idealized zero-divergence case to source entropy. Empirically, the study confirms the trade-off on CIFAR10 and LSUN-Church, demonstrating how reducing noise dimension degrades sample quality (FID/KID) and how network capacity alters the achievable trade-off, with practical implications for compression-aware GAN design.

Abstract

Generative adversial network (GAN) is a type of generative model that maps a high-dimensional noise to samples in target distribution. However, the dimension of noise required in GAN is not well understood. Previous approaches view GAN as a mapping from a continuous distribution to another continous distribution. In this paper, we propose to view GAN as a discrete sampler instead. From this perspective, we build a connection between the minimum noise required and the bits to losslessly compress the images. Furthermore, to understand the behaviour of GAN when noise dimension is limited, we propose divergence-entropy trade-off. This trade-off depicts the best divergence we can achieve when noise is limited. And as rate distortion trade-off, it can be numerically solved when source distribution is known. Finally, we verifies our theory with experiments on image generation.

Noise Dimension of GAN: An Image Compression Perspective

TL;DR

This work reframes GANs as discrete samplers and establishes a direct link between the required noise dimension and the bitrate needed to losslessly encode the source data. It introduces a divergence-entropy trade-off to characterize GAN performance under limited noise and provides a numerical approach (via disciplined convex-concave programming) for known source distributions. Theoretical results bound the minimum noise dimension by per dimension (with refinements for bijective mappings) and connect the idealized zero-divergence case to source entropy. Empirically, the study confirms the trade-off on CIFAR10 and LSUN-Church, demonstrating how reducing noise dimension degrades sample quality (FID/KID) and how network capacity alters the achievable trade-off, with practical implications for compression-aware GAN design.

Abstract

Generative adversial network (GAN) is a type of generative model that maps a high-dimensional noise to samples in target distribution. However, the dimension of noise required in GAN is not well understood. Previous approaches view GAN as a mapping from a continuous distribution to another continous distribution. In this paper, we propose to view GAN as a discrete sampler instead. From this perspective, we build a connection between the minimum noise required and the bits to losslessly compress the images. Furthermore, to understand the behaviour of GAN when noise dimension is limited, we propose divergence-entropy trade-off. This trade-off depicts the best divergence we can achieve when noise is limited. And as rate distortion trade-off, it can be numerically solved when source distribution is known. Finally, we verifies our theory with experiments on image generation.
Paper Structure (12 sections, 2 theorems, 12 equations, 5 figures, 3 tables)

This paper contains 12 sections, 2 theorems, 12 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Preliminary: Generative Adversial Network
  3. Related Works
  4. Noise Dimension Required in GAN
  5. Entropy of IEEE 754 Single-Precision Floating Point Gaussian Distribution
  6. Entropy of Other-Precision Floating Point
  7. Noise Dimension Required in Perfect GAN
  8. Noise Dimension Required in Non-Perfect GAN
  9. Experiments
  10. Experiment Setup
  11. Results and Analysis
  12. Conclusion

Key Result

Lemma 4.1

(Transform reduces entropy) Cover1991ElementsOI Assume $g_{\theta}(.)$ is a deterministic transform, we have with equality holds iff $g_{\theta}(Z)$ is bijective.

Figures (5)

  • Figure 1: (a) Source distribution. (b) $d(\epsilon)$ curve.
  • Figure 2: The result of FID on CIFAR10 and LSUN-Church.
  • Figure 3: BIGGAN's KID on CIFAR10.
  • Figure 4: StyleGAN-ada's KID on CIFAR10 and LSUN-Church.
  • Figure 5: Samples using different noise dim.

Theorems & Definitions (3)

  • Lemma 4.1
  • Theorem 4.2
  • proof