Towards Data-Efficient Language Models: A Child-Inspired Approach to Language Learning

TL;DR

The paper tackles the problem of data-efficient language model training, inspired by human child language acquisition, and contributes a BabyLM-inspired workflow that trains a decoder-only SmolLM with ~125M parameters on a compact corpus (~10M words, later refined, plus TV dialogue). It demonstrates that careful data curation, a 32k token vocabulary, and curriculum learning can yield performance competitive with or surpassing baselines on several benchmarks, while adding TV-rich data and avoiding high-resource corpora like MADLAD can be beneficial or detrimental depending on the dataset. Key findings include improved BLiMP performance with TV data, an optimal ~32k vocabulary size, and curriculum-driven gains, along with a cautionary result that MADLAD data may degrade performance in this low-resource regime. The work highlights practical implications for data-efficient NLP and provides insights into cognitive plausibility for language learning in machines, suggesting avenues for advanced data-valuation and curriculum strategies to further reduce data requirements.

Abstract

In this work, we explain our approach employed in the BabyLM Challenge, which uses various methods of training language models (LMs) with significantly less data compared to traditional large language models (LLMs) and are inspired by how human children learn. While a human child is exposed to far less linguistic input than an LLM, they still achieve remarkable language understanding and generation abilities. To this end, we develop a model trained on a curated dataset consisting of 10 million words, primarily sourced from child-directed transcripts. The 2024 BabyLM Challenge initial dataset of 10M words is filtered to 8.5M. Next, it is supplemented with a randomly selected subset of TVR dataset consisting of 1.5M words of television dialogues. The latter dataset ensures that similar to children, the model is also exposed to language through media. Furthermore, we reduce the vocabulary size to 32,000 tokens, aligning it with the limited vocabulary of children in the early stages of language acquisition. We use curriculum learning and is able to match the baseline on certain benchmarks while surpassing the baseline on others. Additionally, incorporating common LLM training datasets, such as MADLAD-400, degrades performance. These findings underscore the importance of dataset selection, vocabulary scaling, and curriculum learning in creating more data-efficient language models that better mimic human learning processes.