A Distributional Perspective on Word Learning in Neural Language Models

TL;DR

The paper investigates whether neural language models acquire words in human-like order by tracking a broad set of distributional signatures that describe lexical knowledge during training. It generalizes prior surprisal-based metrics by including positive, negative, all-context, intrinsic, and reference-based signatures, and uses a convergence-based AoA extraction method. Experiments train GPT-2–size LMs from scratch on developmentally plausible datasets (Unified, BabyLM, CHILDES) and compare LM AoAs to Wordbank child AoA data, finding that signatures are complementary but largely not aligned with human trajectories. The work provides a framework for richer distributional analysis of word learning in LMs and highlights the need for training regimes and evaluation methods that yield more human-like developmental patterns.

Abstract

Language models (LMs) are increasingly being studied as models of human language learners. Due to the nascency of the field, it is not well-established whether LMs exhibit similar learning dynamics to humans, and there are few direct comparisons between learning trajectories in humans and models. Word learning trajectories for children are relatively well-documented, and recent work has tried to extend these investigations to language models. However, there are no widely agreed-upon metrics for word learning in language models. We take a distributional approach to this problem, defining lexical knowledge in terms of properties of the learned distribution for a target word. We argue that distributional signatures studied in prior work fail to capture key distributional information. Thus, we propose an array of signatures that improve on earlier approaches by capturing knowledge of both where the target word can and cannot occur as well as gradient preferences about the word's appropriateness. We obtain learning trajectories for a selection of small language models we train from scratch, study the relationship between different distributional signatures, compare how well they align with human word learning trajectories and interpretable lexical features, and address basic methodological questions about estimating these distributional signatures. Our metrics largely capture complementary information, suggesting that it is important not to rely on a single metric. However, across all metrics, language models' learning trajectories fail to correlate with those of children.

Figures (6)

• Figure 1: Trajectories for a sample of 8 words for LMs trained on the Unified dataset. We sample one high-frequency (solid line) and one low-frequency (dashed) word from each of the categories: function words, nouns, adjectives, verbs. The $y$-axis represents the value of the estimator in all $\widehat{\sigma\xspace}$ plots. For the children, it represents the proportion of them that produced the word.
• Figure 2: Pearson's correlation coefficients between different signatures and children's $\color{black} AoA$ (C) across three datasets: CHILDES, BabyLM, and Unified.
• Figure A3: Validation losses for models trained on Unified, BabyLM, and CHILDES. The curves show the necessity for an earlier stopping step for seeds 42 (blue), 123 (orange), and 28053 (green).
• Figure A4: Percentage of words that did not converge across various $\epsilon$.
• Figure A5: For each estimator, we present the Pearson correlation coefficient matrix comparing different $\epsilon$. Warmer colors indicate stronger positive correlations, cooler colors indicate stronger negative correlations.