Measuring Grammatical Diversity from Small Corpora: Derivational Entropy Rates, Mean Length of Utterances, and Annotation Invariance

TL;DR

This work develops a principled framework for quantifying syntactic diversity from small corpora by linking mean length of utterances (MLU) to derivational entropy through the derivational entropy rate h[G] = H[G]/MLU[G]. It introduces Smoothed Induced Treebank Entropy (SITE), a bias-corrected estimator that enables accurate entropy estimates from very small treebanks and extends to dependency trees via conversion to derivation trees. The results show that MLU is not merely a proxy but tightly linked to grammatical diversity, with derivational entropy rates largely constant within an annotation convention and across related languages, while SITE reliably converges on homogeneous data and serves as a diagnostic for heterogeneity. These findings have practical implications for NLP and psycholinguistics, enabling theory-free, cross-corpus comparisons of syntactic complexity and informing parsing difficulty and processing studies. The work also proposes a conjecture about consistent grammar families and annotation invariance, suggesting relative entropy rankings remain stable across consistent annotation schemes.

Abstract

In many fields, such as language acquisition, neuropsychology of language, the study of aging, and historical linguistics, corpora are used for estimating the diversity of grammatical structures that are produced during a period by an individual, community, or type of speakers. In these cases, treebanks are taken as representative samples of the syntactic structures that might be encountered. Generalizing the potential syntactic diversity from the structures documented in a small corpus requires careful extrapolation whose accuracy is constrained by the limited size of representative sub-corpora. In this article, I demonstrate -- theoretically, and empirically -- that a grammar's derivational entropy and the mean length of the utterances (MLU) it generates are fundamentally linked, giving rise to a new measure, the derivational entropy rate. The mean length of utterances becomes the most practical index of syntactic complexity; I demonstrate that MLU is not a mere proxy, but a fundamental measure of syntactic diversity. In combination with the new derivational entropy rate measure, it provides a theory-free assessment of grammatical complexity. The derivational entropy rate indexes the rate at which different grammatical annotation frameworks determine the grammatical complexity of treebanks. I introduce the Smoothed Induced Treebank Entropy (SITE) as a tool for estimating these measures accurately, even from very small treebanks. I conclude by discussing important implications of these results for both NLP and human language processing.

Figures (10)

• Figure 1: (a) Example of a syntactic derivation tree as can be contained in a treebank (example taken from the Penn Treebank Marcus:etal:1995). (b) Context-free derivation rules induced from the tree in (a). (c) Example of a dependency tree encoding the syntactic structure of the example in (a-b). (d) Dependency relations induced from the dependency graph in (c). The capitalized symbols are non-terminals and the italicized ones are terminals.
• Figure 2: Representations of the dependency graph in Fig. \ref{['fig:trees']}c as a context-free derivation trees: (a) Omitting the dependency labels. (b) Taking dependency labels into account by means of additional non-terminals.
• Figure 3: Expansion of node $A_i$ with the relations depicted in (a) into the corresponding part context-free derivation tree in (b).
• Figure 4: Results of Corpus Analysis \ref{['sec:c1a']}. Error bars are 95% confidence intervals for the mean. Notice the horizontal logarithmic scales. (a) Estimated derivational entropies according to the four estimators as a function of the sample size. (b) Percentage of rules and non-terminal symbols from the original grammar that are documented as a function of sample size (the 95% C.I.s are imperceptible).
• Figure 5: Results of Corpus Analysis \ref{['sec:c1b']}. Error bars are 95% confidence intervals for the mean. Notice the horizontal logarithmic scales. (a) Estimated derivational entropies according to the four estimators as a function of the sample size. (b) Percentage of rules and non-terminal symbols from the original grammar that are documented as a function of sample size (the 95% C.I.s are imperceptible).