BabyBabelLM: A Multilingual Benchmark of Developmentally Plausible Training Data
Jaap Jumelet, Abdellah Fourtassi, Akari Haga, Bastian Bunzeck, Bhargav Shandilya, Diana Galvan-Sosa, Faiz Ghifari Haznitrama, Francesca Padovani, Francois Meyer, Hai Hu, Julen Etxaniz, Laurent Prévot, Linyang He, María Grandury, Mila Marcheva, Negar Foroutan, Nikitas Theodoropoulos, Pouya Sadeghi, Siyuan Song, Suchir Salhan, Susana Zhou, Yurii Paniv, Ziyin Zhang, Arianna Bisazza, Alex Warstadt, Leshem Choshen
TL;DR
BabyBabelLM addresses the need for data-efficient, developmentally plausible multilingual pretraining by providing a 45-language dataset with three data-budget tiers anchored to English tokens ($100$M,$10$M,$1$M). It combines CDS, education, books, wikis, and subtitles within a structured preprocessing pipeline and accompanies a scalable, multilingual evaluation suite spanning formal and functional competence. The work demonstrates baseline monolingual and multilingual experiments using GPT-2–style architectures and highlights data-efficiency, cross-linguistic transfer, and the benefits and limitations of bilingual inputs. Overall, BabyBabelLM aims to enable reproducible, comparable, and inclusive research on language acquisition and multilingual learning, with an open invitation for community expansion and richer multilingual evaluation.
Abstract
We present BabyBabelLM, a multilingual collection of datasets modeling the language a person observes from birth until they acquire a native language. We curate developmentally plausible pretraining data aiming to cover the equivalent of 100M English words of content in each of 45 languages. We compile evaluation suites and train baseline models in each language. BabyBabelLM aims to facilitate multilingual pretraining and cognitive modeling.