How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model

TL;DR

The paper introduces the Random Hierarchy Model (RHM) to study how deep neural networks learn hierarchical, compositional data and to quantify the required number of examples. It shows shallow networks face the curse of dimensionality with a data requirement scaling as $P_{ ext{max}} = n_c m^{(d-1)/(s-1)}$ while deep networks achieve a polynomial $P^* \approx n_c m^L$, with $d = s^L$, highlighting depth as a mechanism to harness structure. The authors connect this efficiency to the emergence of synonymic invariances and detectable local correlations between low-level features and labels, arguing that $P_c \approx P^*$ and that invariances reduce effective dimensionality; clustering can further improve sample efficiency. They demonstrate how removing correlations reintroduces the curse, underscoring the critical role of data structure in learnability and offering a framework to estimate data needs for hierarchical tasks. Overall, the work provides a quantitative, testable account of why depth helps in representation learning and suggests avenues for extending these ideas to gradient dynamics and self-supervised learning.

Abstract

Deep learning algorithms demonstrate a surprising ability to learn high-dimensional tasks from limited examples. This is commonly attributed to the depth of neural networks, enabling them to build a hierarchy of abstract, low-dimensional data representations. However, how many training examples are required to learn such representations remains unknown. To quantitatively study this question, we introduce the Random Hierarchy Model: a family of synthetic tasks inspired by the hierarchical structure of language and images. The model is a classification task where each class corresponds to a group of high-level features, chosen among several equivalent groups associated with the same class. In turn, each feature corresponds to a group of sub-features chosen among several equivalent ones and so on, following a hierarchy of composition rules. We find that deep networks learn the task by developing internal representations invariant to exchanging equivalent groups. Moreover, the number of data required corresponds to the point where correlations between low-level features and classes become detectable. Overall, our results indicate how deep networks overcome the curse of dimensionality by building invariant representations, and provide an estimate of the number of data required to learn a hierarchical task.

Figures (15)

• Figure 1: The Random Hierarchy Model. Left: Structure of the generative model. The class label $\alpha\,{=}\,1,\dots,n_c$ generates a set of $m$ equivalent (i.e., synonymic) high-level representations with elements taken from a vocabulary of high-level features $\mathcal{V}_L$. Similarly, high-level features generate $m$ equivalent lower-level representations, taken from a vocabulary $\mathcal{V}_{L-1}$. Repeating this procedure $L\,{-}\,2$ times yields all the input data with label $\alpha$, consisting of low-level features taken from $\mathcal{V}_1$. Right: example of Random Hierarchy Model with $n_c\,{=}\,2$ classes, $L\,{=}\,3$, $s\,{=}\,2$, $m\,{=}\,3$ and homogeneous vocabulary size $v_1\,{=}\,v_2\,{=}\,v_3\,{=}\,3$. The three sets of rules are listed at the top, while two examples of data generation are shown at the bottom. The first example is obtained by following the rules in the colored boxes.
• Figure 2: Sample complexity of two-layer fully-connected networks, for $L=s\,{=}\,2$ and $v\,{=}\,n_c\,{=}\,m$. Top: Test error vs the number of training data. Different colors correspond to different vocabulary sizes $v$. Bottom: number of training data resulting in test error $\bar{\epsilon}\,{=}\,0.7$ as a function of $P_{\text{max}}$, with the black dashed line indicating a linear relationship.
• Figure 3: Sample complexity of depth-$(L+1)$ CNNs, for $s\,{=}\,2$ and $m\,{=}\,n_c\,{=}\,v$. Top: Test error vs number of training points. Different colors correspond to different vocabulary sizes $v$ while the markers indicate the hierarchy depth $L$. Bottom: sample complexity $P^*$ corresponding to a test error $\epsilon^*=0.1 \epsilon_{\text{rand}}$. The empirical points show remarkable agreement with the law $P^* = n_c m^{L}$, shown as a black dashed line.
• Figure 4: Sample complexity of depth-$(L+1)$ CNNs, for $s\,{=}\,2$, $n_c\,{=}\,v$ and varying $m\,{\leq}\,v$. Top: Test error vs number of training points, with different colors corresponding to different vocabulary sizes $v$ and markers indicating the hierarchy depth $L$. Bottom: sample complexity $P^*$, with the law $P^* = n_c m^L$ shown as a black dashed line.
• Figure 5: Synonymic sensitivity $S_{2, 1}$ for a depth-$(L+1)$ CNN trained on the RHM with $s=2$, $n_c=m=v$ as a function of the training set size ($L$ and $v$ as in the key). The collapse achieved after rescaling by $P^* = n_c m^L$ highlights that the sample complexity coincides with the number of training points required to build internal representations invariant to exchanging synonyms.