Learning curves theory for hierarchically compositional data with power-law distributed features

TL;DR

The paper investigates how hierarchical compositional structure and Zipf-distributed features shape learning curves under data-limited regimes. By integrating the Random Hierarchy Model with Zipf feature frequencies into a PCFG framework, it derives that classification learning curves become power laws with exponent $a/(1+a)$, while next-token prediction’s asymptotic exponent depends on the hierarchy but not on $a$. The authors support these predictions with theoretical analyses and empirical experiments using deep CNNs and Transformers on RHM-generated data, showing pre-asymptotic phases governed by hierarchy and asymptotic behavior set by rule distributions. The work provides a principled link between data geometry, hierarchical generation, and scaling laws, offering insights into why language models exhibit certain scaling patterns and how architecture choice (CNNs vs. Transformers) interacts with data structure.

Abstract

Recent theories suggest that Neural Scaling Laws arise whenever the task is linearly decomposed into power-law distributed units. Alternatively, scaling laws also emerge when data exhibit a hierarchically compositional structure, as is thought to occur in language and images. To unify these views, we consider classification and next-token prediction tasks based on probabilistic context-free grammars -- probabilistic models that generate data via a hierarchy of production rules. For classification, we show that having power-law distributed production rules results in a power-law learning curve with an exponent depending on the rules' distribution and a large multiplicative constant that depends on the hierarchical structure. By contrast, for next-token prediction, the distribution of production rules controls the local details of the learning curve, but not the exponent describing the large-scale behaviour.

Figures (8)

• Figure 1: Pictorial representation of a derivation according to the RHM, with depth $L\,{=}\,3$ and branching factor $s\,{=}\,2$. A classification task requires predicting the root label (blue square) from the leaves. The correlations between the $2$-tuples of leaves (e.g. $(x_5,x_6)$) and the label $y$ can be used to infer the hidden symbol above the $2$-tuple ($\mu^{ (1)}_3$ for $(x_5,x_6)$). A next-token prediction task requires predicting the last observable symbol (red square) from the previous $d\,{-}\,1$. In this case, hidden symbols can be deduced from the correlations of $2$-tuples with the last token $x_d$.
• Figure 2: Left: Learning curves of $3$-layers CNNs trained on RHM data with $L\,{=}\,2$, $s\,{=}\,2$, $v\,{=}\,m\,{=}\,25$ and Zipf exponent $a$ indicated in caption. Solid lines are the empirical learning curves whereas dotted lines are predictions from Eq. (\ref{['eq:class-learning-curve']}). The dashed line represents the scaling law $\epsilon\sim P^{-a/(1+a)}$. Right: As in the left panel, but $v\,{=}\,m\,{=}\,100$. Here $a$ is fixed and the layer where production rules are Zipf-distributed changes. The black dotted line represents the scaling law $\epsilon\sim P^{-a/(1+a)}$.
• Figure 3: Left: Learning curves in the same setting as \ref{['fig:class-empirical']}, with Zipf exponent $a\,{=}\,1$ and $m=v$ indicated in caption. Solid lines are the empirical learning curves whereas dotted lines are predictions from Eq. (\ref{['eq:class-learning-curve']}). Right: all the curves collapse when rescaling the x-axis by $v m^{L-1}$---the sample complexity of an RHM with uniform production rules and $L\,{-}\,1$ layers. The black dotted line represents the scaling law $\epsilon\sim P^{-a/(1+a)}$.
• Figure 4: Left: Empirical learning curve of one-layer transformers trained for next-token prediction on RHM data with $L\,{=}\,1$, $s\,{=}\,2$, $v\,{=}\,128$, $m\,{=}\,32$ and Zipf exponent $a$ as in the key. Vertical dashed lines mark the sample sizes required to learn the most frequent rules: $vm$ in the uniform case (equivalent to setting $a\,{=}\,-1$ in Zipf's law) and $v$ with Zipf-distributed production rules. The leftmost horizontal dashed lines denote the test loss of the trivial prediction where the next-token probability is uniform over the vocabulary, $\mathcal{L}_0\,{=}\,\log{v}$. The rightmost horizontal dashed lines denote the average cross-entropy of the $s$-gram approximation, $\mathcal{L}_1(a)$. Right: subtracting $\mathcal{L}_1(a)$ reveals the power-law decay $P^{-a/(1+a)}$, highlighted by coloured dashed lines.
• Figure 5: Average cross-entropies of the $s^\ell$-grams versus $\ell$, for RHM datasets with $s\,{=}\,2$, $v\,{=}\,32$, $m\,{=}\,8$, with the colour denoting the Zipf exponent. The points are obtained by averaging the cross-entropies over $32$ independent realisations of the RHM. The cross-entropies of the uniform production rules case are shown in blue for comparison. For all $a$'s, the cross-entropies $\mathcal{L}_\ell$ decay with $\ell$ towards some $a$-dependent value $\mathcal{L}_\infty(a)$ (Top panel). However, the approach to $\mathcal{L}_\infty(a)$ is independent of $a$ (Bottom panel) and follows the behaviour of the test loss bound derived in cagnetta2024towards in the uniform case (black dashed line).