Provable Learning of Random Hierarchy Models and Hierarchical Shallow-to-Deep Chaining

TL;DR

The paper proves that deep, gradient-trained convolutional networks can provably exploit hierarchical structure in Random Hierarchy Models (RHMs), a class of hierarchical probabilistic grammars. By formalizing a shallow-to-deep chaining principle and employing layerwise gradient descent with random Fourier feature mappings, it achieves end-to-end learning with sample complexity $O(m^{(1+o(1))L})$, matching the deep-depth conjecture while outperforming shallow-depth limits of $O(m^{s^L})$ in the regime $d = s^L$. The approach hinges on three core conditions—correlation between target and lower-level outputs, clean signal transfer to lower layers, and weak identifiability of lower-level features—and demonstrates how these enable hierarchical learning through successive layerwise optimizations. The results rely on a careful analysis of the RF layer, concentration bounds, and a recursive error-control scheme to ensure the learned proxies at each level faithfully guide subsequent layers. This work advances theoretical understanding of depth efficiency in hierarchical learning and points to broader applicability of the shallow-to-deep chaining framework in other structured target classes.

Abstract

The empirical success of deep learning is often attributed to deep networks' ability to exploit hierarchical structure in data, constructing increasingly complex features across layers. Yet despite substantial progress in deep learning theory, most optimization results sill focus on networks with only two or three layers, leaving the theoretical understanding of hierarchical learning in genuinely deep models limited. This leads to a natural question: can we prove that deep networks, trained by gradient-based methods, can efficiently exploit hierarchical structure? In this work, we consider Random Hierarchy Models -- a hierarchical context-free grammar introduced by arXiv:2307.02129 and conjectured to separate deep and shallow networks. We prove that, under mild conditions, a deep convolutional network can be efficiently trained to learn this function class. Our proof builds on a general observation: if intermediate layers can receive clean signal from the labels and the relevant features are weakly identifiable, then layerwise training each individual layer suffices to hierarchically learn the target function.

Figures (3)

• Figure 1: Tree description of a $3$-level RHM instance with branching factor $3$. Each node is a random variable. The root node is the label and the leaves are the input sequence. At level $l$, the condition probability of the children patch being $\bm{\mu} \in \mathcal{V}_{l+1}^s$ conditioned on the parent token being $\nu \in \mathcal{V}_l$ is given by $\mathcal{Q}_l(\nu, \bm{\mu})$.
• Figure 2: One layer of our learner model. For each level $l$, the model acts on each patch $(\bm{h}^{(l)}_{(k-1)s}, \dots, \bm{h}^{(l)}_{ks})$ independently using the same set of weights ${\bm{W}}^{(l)} \in \mathbb{R}^{d_y \times d_x}$ and nonlinearity $\bm{\Phi}^{(l)}: \mathbb{R}^{s d_e} \to \mathbb{R}^{d_e}$. The output on the $k$-th level-$l$ patch will be used as the embedding of $k$-th token at level $l-1$.
• Figure 3: Example of a deep quadratic function. The target function is the sum of all terms appearing in the graph. That is, $f({\bm{x}}) = x_1 \cdots x_8 + x_9 \cdots x_{12} + x_1 \cdots x_4 + x_5 \cdots x_8 + x_1x_2 + \cdots x_7 x_8$.

Theorems & Definitions (48)

• Theorem 1.1: Informal version of Theorems \ref{['thm: opt main']}
• Definition 2.1: Random Hierarchy Model
• Definition 2.2: CFG-induced RHM
• proof : Remark
• Theorem 4.0
• proof : Remark
• Lemma 4.0: Error propagation in the random features layer
• Lemma 4.0: Training a single layer
• Lemma A.0: Error propagation in the random features layer
• proof