How Well Can Generative Adversarial Networks Learn Densities: A Nonparametric View
Tengyuan Liang
TL;DR
This work frames GANs as nonparametric density estimators and derives rates of convergence that adapt to the smoothness of the target density. By introducing a regularized/smoothed empirical measure, the authors obtain a rate of $n^{-(\alpha+\beta)/(2(\alpha+\beta)+d)}$ for the excess GAN risk, and prove a near-matching minimax lower bound, showing near-optimality in high dimensions. They extend the theory to neural-network realizations, demonstrating that deep ReLU discriminators and generators can achieve these rates under suitable approximation properties, with explicit improvements for deeper discriminators. A key byproduct is improved generalization bounds for GANs with deep discriminators, clarifying how smoothness and architecture impact learning performance in density estimation. Overall, the paper provides a quantitative, nonparametric foundation for understanding how well GANs can learn a wide class of densities across different evaluation metrics.
Abstract
We study in this paper the rate of convergence for learning densities under the Generative Adversarial Networks (GAN) framework, borrowing insights from nonparametric statistics. We introduce an improved GAN estimator that achieves a faster rate, through simultaneously leveraging the level of smoothness in the target density and the evaluation metric, which in theory remedies the mode collapse problem reported in the literature. A minimax lower bound is constructed to show that when the dimension is large, the exponent in the rate for the new GAN estimator is near optimal. One can view our results as answering in a quantitative way how well GAN learns a wide range of densities with different smoothness properties, under a hierarchy of evaluation metrics. As a byproduct, we also obtain improved generalization bounds for GAN with deeper ReLU discriminator network.