Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bringing Up a Bilingual BabyLM: Investigating Multilingual Language Acquisition Using Small-Scale Models

Linda Zeng, Steven Y. Feng, Michael C. Frank

Abstract

Multilingualism is incredibly common around the world, leading to many important theoretical and practical questions about how children learn multiple languages at once. For example, does multilingual acquisition lead to delays in learning? Are there better and worse ways to structure multilingual input? Many correlational studies address these questions, but it is surprisingly difficult to get definitive answers because children cannot be randomly assigned to be multilingual and data are typically not matched between languages. We use language model training as a method for simulating a variety of highly controlled exposure conditions, and create matched 100M-word mono- and bilingual datasets using synthetic data and machine translation. We train GPT-2 models on monolingual and bilingual data organized to reflect a range of exposure regimes, and evaluate their performance on perplexity, grammaticality, and semantic knowledge. Across model scales and measures, bilingual models perform similarly to monolingual models in one language, but show strong performance in the second language as well. These results suggest that there are no strong differences between different bilingual exposure regimes, and that bilingual input poses no in-principle challenges for agnostic statistical learners.

Bringing Up a Bilingual BabyLM: Investigating Multilingual Language Acquisition Using Small-Scale Models

Abstract

Multilingualism is incredibly common around the world, leading to many important theoretical and practical questions about how children learn multiple languages at once. For example, does multilingual acquisition lead to delays in learning? Are there better and worse ways to structure multilingual input? Many correlational studies address these questions, but it is surprisingly difficult to get definitive answers because children cannot be randomly assigned to be multilingual and data are typically not matched between languages. We use language model training as a method for simulating a variety of highly controlled exposure conditions, and create matched 100M-word mono- and bilingual datasets using synthetic data and machine translation. We train GPT-2 models on monolingual and bilingual data organized to reflect a range of exposure regimes, and evaluate their performance on perplexity, grammaticality, and semantic knowledge. Across model scales and measures, bilingual models perform similarly to monolingual models in one language, but show strong performance in the second language as well. These results suggest that there are no strong differences between different bilingual exposure regimes, and that bilingual input poses no in-principle challenges for agnostic statistical learners.

Paper Structure

This paper contains 23 sections, 4 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Methods
  4. Training Data
  5. Training Conditions
  6. Model Training
  7. Evaluation
  8. Additional Experiments
  9. Results
  10. Main Experiment
  11. Varying Exposure
  12. Reduced Model and Data Size
  13. Qualitative Analysis and Model Interpretation
  14. Discussion
  15. Conclusion
  16. ...and 8 more sections

Figures (4)

  • Figure 1: General motivation and condition structure for main experiments. Note "by-speaker” refers to a one speaker–one language policy (shown in the example), while “random” allows either speaker to produce either language (randomly assigned per dialogue).
  • Figure 2: Multilingual and cross-lingual word similarity (X-WS) by training condition for GPT-2 models (124M). Dashes indicate the mean score across three training seeds.
  • Figure 3: Evaluation results for GPT-2 (124M) trained with varying language proportions in the Multilingual (random) condition. Markers denote the means over three seeds.
  • Figure 4: Two-dimensional visualizations of the trained GPT-2 (124M) token embedding space, with English, Spanish, and shared tokens in blue, red, and purple respectively.