Is Child-Directed Speech Effective Training Data for Language Models?

TL;DR

These findings support the hypothesis that, rather than proceeding from better data, the child’s learning algorithm is substantially more data-efficient than current language modeling techniques.

Abstract

While high-performing language models are typically trained on hundreds of billions of words, human children become fluent language users with a much smaller amount of data. What are the features of the data they receive, and how do these features support language modeling objectives? To investigate this question, we train GPT-2 and RoBERTa models on 29M words of English child-directed speech and a new matched, synthetic dataset (TinyDialogues), comparing to OpenSubtitles, Wikipedia, and a heterogeneous blend of datasets from the BabyLM challenge. We evaluate the syntactic and semantic knowledge of these models using developmentally-inspired evaluations. Through pretraining experiments, we test whether the global developmental ordering or the local discourse ordering of children's training data supports high performance relative to other datasets. The local properties of the data affect model results, but surprisingly, global properties do not. Further, child language input is not uniquely valuable for training language models. These findings support the hypothesis that, rather than proceeding from better data, the child's learning algorithm is substantially more data-efficient than current language modeling techniques.

Figures (8)

• Figure 1: Total CHILDES word counts (utterances only, no metadata) by age.
• Figure 2: GPT-2 convergence graphs (train and val loss) by dataset, using iterative training for 20 epochs. From top to bottom: CHILDES, TinyDialogues, BabyLM.
• Figure 3: GPT-2 convergence graphs (train and val loss) of TinyDialogues using the typical iterative training approach for 20 epochs, for different global orders. From top to bottom: age order, reverse order, random order.
• Figure 4: GPT-2 convergence graphs (train and val loss) of CHILDES using the repeated buckets training approach with $b=5, n=10$, for different global orders. From top to bottom: age, reverse, random order.
• Figure 5: GPT-2 convergence graphs (train and val loss) of TinyDialogues using the repeated buckets training approach with $n=10$, for different global orders. From top to bottom: age order, reverse order, random order.