Rethinking the Relationship between the Power Law and Hierarchical Structures

TL;DR

This paper interrogates the widely cited connection between power-law decay of correlation in language and underlying hierarchical structures. It empirically tests three core assumptions using parse trees from English (BLLIP, WikiText) and Japanese (NPCMJ). It shows that, in natural languages, correlation decays with structural distance according to a power law rather than exponentially, that sequential distance grows sub-exponentially with structural distance due to strong branching, and that natural-language syntax deviates from PCFGs as measured by CFIB. The findings challenge the lin2017critical argument and suggest exploring discourse-level hierarchical organization or alternative mechanisms, with implications for linguistic theory and language-model design.

Abstract

Statistical analysis of corpora provides an approach to quantitatively investigate natural languages. This approach has revealed that several power laws consistently emerge across different corpora and languages, suggesting universal mechanisms underlying languages. Particularly, the power-law decay of correlation has been interpreted as evidence for underlying hierarchical structures in syntax, semantics, and discourse. This perspective has also been extended to child speeches and animal signals. However, the argument supporting this interpretation has not been empirically tested in natural languages. To address this problem, the present study examines the validity of the argument for syntactic structures. Specifically, we test whether the statistical properties of parse trees align with the assumptions in the argument. Using English and Japanese corpora, we analyze the mutual information, deviations from probabilistic context-free grammars (PCFGs), and other properties in natural language parse trees, as well as in the PCFG that approximates these parse trees. Our results indicate that the assumptions do not hold for syntactic structures and that it is difficult to apply the proposed argument to child speeches and animal signals, highlighting the need to reconsider the relationship between the power law and hierarchical structures.

Figures (14)

• Figure 1: Sequential and structural distances. (a) If trees are balanced, the sequential distance grows exponentially with the structural distance, $r_{\text{seq}} \sim \exp ( \mu r_{\text{str}})$. (b) If trees are strongly biased, the growth is slower. For example, the relation is linear, $r_{\text{seq}} \sim r_{\text{str}}$.
• Figure 2: Example of a parse tree from BLLIP, preprocessed as described in Section \ref{['subsec_dataset']}. This tree has a strong right-branching bias.
• Figure 3: CFIB $J$ is the MI between the children of two nodes whose categories are fixed.
• Figure 4: Estimation and fitting of the MI for (a) the exponential model with $\lambda = 0.1$ and (b) the power-law model with $\alpha = 2$. Fitting was performed for $N_{\text{data}} = 8 \times 10^6$.
• Figure 5: MI $I_{\text{POS}}$ between POS tags as a function of the sequential distance $r_{\text{seq}}$. The fitted exponential and power-law decays for $N_{\text{data}} = 2.56 \times 10^6$ are also presented.