Deep Networks Learn Deep Hierarchical Models
Amit Daniely
TL;DR
The paper investigates why deep networks, specifically layerwise SGD on residual architectures, can efficiently learn hierarchical multi-label tasks when the target function exhibits an unknown label hierarchy $L_1 \subseteq \dots \subseteq L_r = [n]$ with simple first-level labels and higher levels formed from simpler ones. It formalizes polynomial-threshold-function (PTF) hierarchies and introduces a brain-dump-inspired model where auxiliary random labels render the hierarchy learnable; a residual-network training procedure is shown to achieve polynomial-time learnability with poly-size networks given polynomially many samples. The main contribution is a rigorous demonstration that very deep hierarchical structures are within the reach of practical gradient-based optimization, extending prior results that were restricted to log-depth circuits, and providing a foundational perspective on how granular supervision could shape deep learning. The work also provides a suite of technical tools—PTFs, Hermite analyses, kernel/random-feature theory, and concentration results—that may guide future theoretical and empirical exploration of hierarchical learning in neural networks.
Abstract
We consider supervised learning with $n$ labels and show that layerwise SGD on residual networks can efficiently learn a class of hierarchical models. This model class assumes the existence of an (unknown) label hierarchy $L_1 \subseteq L_2 \subseteq \dots \subseteq L_r = [n]$, where labels in $L_1$ are simple functions of the input, while for $i > 1$, labels in $L_i$ are simple functions of simpler labels. Our class surpasses models that were previously shown to be learnable by deep learning algorithms, in the sense that it reaches the depth limit of efficient learnability. That is, there are models in this class that require polynomial depth to express, whereas previous models can be computed by log-depth circuits. Furthermore, we suggest that learnability of such hierarchical models might eventually form a basis for understanding deep learning. Beyond their natural fit for domains where deep learning excels, we argue that the mere existence of human ``teachers" supports the hypothesis that hierarchical structures are inherently available. By providing granular labels, teachers effectively reveal ``hints'' or ``snippets'' of the internal algorithms used by the brain. We formalize this intuition, showing that in a simplified model where a teacher is partially aware of their internal logic, a hierarchical structure emerges that facilitates efficient learnability.