Deep Generative Models: Complexity, Dimensionality, and Approximation

TL;DR

This work provides a theoretical framework showing that deep generative models can approximate distributions on $d$-dimensional Riemannian manifolds from inputs of arbitrary dimension $m\ge 1$, even when $m<d$, by leveraging space-filling curve concepts. It establishes a concrete complexity-accuracy trade-off: as input dimension decreases below the manifold dimension, the required network width grows super-exponentially, revealing a fundamental limitation and guiding practical choices for input dimension. The main results extend previous $d+1$-dimensional constructions to arbitrary $m$, with population and empirical guarantees in the Wasserstein metric, and they are complemented by toy simulations illustrating space-filling behavior. The findings have implications for understanding the practical success of generative models and inform architecture design and dimension estimation in high-dimensional data settings.

Abstract

Generative networks have shown remarkable success in learning complex data distributions, particularly in generating high-dimensional data from lower-dimensional inputs. While this capability is well-documented empirically, its theoretical underpinning remains unclear. One common theoretical explanation appeals to the widely accepted manifold hypothesis, which suggests that many real-world datasets, such as images and signals, often possess intrinsic low-dimensional geometric structures. Under this manifold hypothesis, it is widely believed that to approximate a distribution on a $d$-dimensional Riemannian manifold, the latent dimension needs to be at least $d$ or $d+1$. In this work, we show that this requirement on the latent dimension is not necessary by demonstrating that generative networks can approximate distributions on $d$-dimensional Riemannian manifolds from inputs of any arbitrary dimension, even lower than $d$, taking inspiration from the concept of space-filling curves. This approach, in turn, leads to a super-exponential complexity bound of the deep neural networks through expanded neurons. Our findings thus challenge the conventional belief on the relationship between input dimensionality and the ability of generative networks to model data distributions. This novel insight not only corroborates the practical effectiveness of generative networks in handling complex data structures, but also underscores a critical trade-off between approximation error, dimensionality, and model complexity.

Figures (15)

• Figure 1: Two cases demonstrating the idea of how sufficiently large neural networks can learn distributions of higher dimension than their input sampling distributions by filling out the space. Depicted here are 1-dimensional partial space-filling curves beginning to "fill out" a 2-dimensional unit square (left) and a 2-dimensional cylindrical surface (right). The blue points represent the training sample from the target distribution used to fit the neural network, while the orange points represent new points generated by the trained network.
• Figure 2: Simulation 1: Training results (left) of a small 2 hidden layer, 10 node each, fully connected network mapping the 2-D uniform input to a 2-D uniform target data manifold. The orange surface is generated by the neural network and is "filling in" the data manifold (blue). Wasserstein loss (top right) and fill distance (bottom right) between generated and observed data per training iteration for a total of 10,000 iterations.
• Figure 3: Simulation 1: Training results of a 5 hidden layer, 200 node each, fully connected network for 10,000 iterations. Training trajectory (left), Wasserstein loss (top right), and fill distance (bottom right) of mapping a uniform distribution on the interval $[0,1]$ to a uniform distribution on the unit square $[0,1]^2$. It can be seen that as the loss decreases, the fitted curve fills more of the square as expected.
• Figure 4: Simulation 2: Training results of a small 3 hidden layer, 25 node each, fully connected network for 8,000 iterations mapping a 2-D uniform input to a 2-D uniform target data manifold on a cylinder embedded in $\mathbb{R}^3$. Here the multicolored surface is generated by the neural network (colored by $z$-axis height) and is "filling in" the data manifold (blue).
• Figure 5: Simulation 2: Training results of a 7 hidden layer, 250 node each, fully connected network over 5,000 iterations. Training trajectory (left), Wasserstein loss (top right), and fill distance (bottom right) of mapping a 1-D uniform distribution on the interval $[0,1]$ to a 2-D uniform distribution on a cylinder embedded in $\mathbb{R}^3$. As the loss decreases over the iterations, the fitted curve fills more of the surface.

Theorems & Definitions (18)

• Definition 2.1
• Definition 2.2
• Lemma 2.5: Theorem 1 of dahal2022deep
• Lemma 2.6: Theorem 2 of dahal2022deep
• Theorem 3.1: Approximation Power of Deep Generative Models
• Theorem 3.2: Statistical Guarantees of Deep Generative Models
• Corollary 3.3
• Lemma 5.1
• Lemma 5.2
• Lemma 5.3