Dimension-independent rates for structured neural density estimation

TL;DR

A novel justification for deep learning's ability to circumvent the curse of dimensionality is provided, demonstrating dimension-independent convergence rates in these contexts of image, sound, video, and text data.

Abstract

We show that deep neural networks achieve dimension-independent rates of convergence for learning structured densities such as those arising in image, audio, video, and text applications. More precisely, we demonstrate that neural networks with a simple $L^2$-minimizing loss achieve a rate of $n^{-1/(4+r)}$ in nonparametric density estimation when the underlying density is Markov to a graph whose maximum clique size is at most $r$, and we provide evidence that in the aforementioned applications, this size is typically constant, i.e., $r=O(1)$. We then establish that the optimal rate in $L^1$ is $n^{-1/(2+r)}$ which, compared to the standard nonparametric rate of $n^{-1/(2+d)}$, reveals that the effective dimension of such problems is the size of the largest clique in the Markov random field. These rates are independent of the data's ambient dimension, making them applicable to realistic models of image, sound, video, and text data. Our results provide a novel justification for deep learning's ability to circumvent the curse of dimensionality, demonstrating dimension-independent convergence rates in these contexts.

Figures (8)

• Figure 1: (a): Examples of conditioning $X = x$ (dotted lines) when a density's support is a manifold (solid lines). (b): Highly simplified examples of common MRF graphs. Paths correspond to sequential data and grids to spatial.
• Figure 2: An example MRF. The random variables ${\textnormal{x}}$ and ${\textnormal{y}}$ are independent given ${\textnormal{z}}$.
• Figure 3: Top row: Scatterplots comparing the grayscale values of pixel (8,8) with various other pixels for 100 randomly selected images. The decreasing correlation between pixels as their distance increases is evident. Bottom row: The same comparisons as the top row, but conditioned on pixel (9,8) having a value approximately equal to 0.48 (the median value for this pixel across the dataset). Note the increased concentration of points towards the center along the horizontal axis, indicating reduced correlation when conditioned on a neighboring pixel. These plots demonstrate how pixel correlations decrease with distance and how conditioning on a neighboring pixel can significantly reduce correlations, supporting the use of Markov Random Field models for image data. Similar plots for the COCO dataset can be found in Appendix \ref{['appx:coco']}.
• Figure 4: The leftmost image (a) is a $640 \times 427$ pixel photograph from the COCO 2014 dataset coco. Image (b) shows an enlarged version of the $102 \times 102$ pixel region outlined in (a). Images (c) and (d) display the 12-pixel and 1-pixel width borders of that region, respectively. Modeling this image with an MRF graph $L_{640 \times 427}$ or $L^+_{640 \times 427}$ would imply that the distribution of the missing interior in (d) depends exclusively on its 1-pixel wide border, with the rest of the image in (a) being uninformative for predicting this interior region. In contrast, predicting the interior using the 12-pixel border in (c) is more reasonable. This scenario corresponds to models like $L_{640 \times 427}^6$ or $\left(L^+_{640 \times 427}\right)^6$, which capture more extensive local dependencies. It's important to note that for the MRF model to hold, the interior doesn't need to be deterministically constructed from the surrounding pixels. Rather, the surrounding pixels need only provide sufficient information about the interior (e.g., that it's a cat's face) such that the rest of the image doesn't contribute any additional information for predicting the interior region.
• Figure 5: Illustrations of a path graph and its powers. Left: The path graph $L_5$. Center: The power graph $L_5^2$. Right: The power graph $L_5^3$. In $L_5$, only immediately contiguous vertices are connected. In $L_5^2$, every group of three contiguous vertices forms a complete subgraph. In $L_5^3$, every group of four contiguous vertices forms a complete subgraph. This progression demonstrates increasing connectivity among nearby vertices in the graph.

Theorems & Definitions (33)

• Proposition 4.1: hammersley1971markov
• Theorem 4.2
• Lemma 4.3
• Lemma 4.4
• Corollary 4.5: Dimension-independent rates
• Lemma 4.6
• Lemma 4.7
• Theorem 4.8
• Proposition A.1
• Lemma A.2