From Linear Input to Hierarchical Structure: Function Words as Statistical Cues for Language Learning

TL;DR

This study investigates how function words, through three cross-linguistic properties (high frequency, reliable structural association, and phrase-boundary alignment), anchor the learning of hierarchical syntax. Using counterfactual language modeling with transformer models and rigorous probing/ablation, the authors show a hierarchical influence among these cues (frequency > structural association > boundary alignment) and reveal a Goldilocks regime for function-word distributions. The cross-linguistic UD analysis confirms these properties are broadly universal across 186 languages, while a controlled English-language modeling suite demonstrates that preserving all three cues yields the best learnability, though different internal mechanisms can produce similar performance. The work situates function words as culturally evolved anchors that facilitate segmentation and labeling during grammar acquisition, and highlights the importance of combining behavioral performance with representation-level analyses to understand learning.

Abstract

What statistical conditions support learning hierarchical structure from linear input? In this paper, we address this question by focusing on the statistical distribution of function words. Function words have long been argued to play a crucial role in language acquisition due to their distinctive distributional properties, including high frequency, reliable association with syntactic structure, and alignment with phrase boundaries. We use cross-linguistic corpus analysis to first establish that all three properties are present across 186 studied languages. Next, we use a combination of counterfactual language modeling and ablation experiments to show that language variants preserving all three properties are more easily acquired by neural learners, with frequency and structural association contributing more strongly than boundary alignment. Follow-up probing and ablation analyses further reveal that different learning conditions lead to systematically different reliance on function words, indicating that similar performance can arise from distinct internal mechanisms.

Figures (6)

• Figure 1: Distributional properties of function words across languages.
• Figure 2: An example dependency tree with constituent spans highlighted using colored brackets: NP, VP, and PP. Phrase labels are theoretically neutral.
• Figure 3: The boundary ratio of function and content words across languages.
• Figure 4: Distribution of dominant function heads across BLiMP categories under different training conditions (seed=53). Numbers indicate the top-5 heads and the number of BLiMP categories in which each head assigns the highest attention to function words. Results for other seeds are reported in Appendix \ref{['fig:attresults-other-seeds']}.
• Figure 5: Difference in BLiMP accuracy before and after ablation.