Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A surprisal oracle for when every layer counts

Xudong Hong, Sharid Loáiciga, Asad Sayeed

TL;DR

The paper addresses training language models under data-constrained BabyLM conditions by introducing a dynamic, learner-directed curriculum (ACLM) that recomputes a surprisal space at every training iteration using the current model (ELC-BERT). It evaluates ACLM on the BabyLM 2024 strict-small task, showing that while the approach does not consistently outperform the strongest baselines on fine-grained grammatical tasks, it yields improvements on world-knowledge (EWOK) and GLUE tasks, suggesting benefits from learning-order dynamics. The contributions include a dynamic surprisal-space update mechanism, 128-token sequence processing, and a kNN-based uncertainty-driven curriculum without human-in-the-loop, highlighting the cognitive plausibility of learner-directed training. The work indicates that ACLM can augment high-performing architectures in data-limited regimes and points to future work on hyperparameter exploration and alternative uncertainty signals to further unlock learner-like language acquisition dynamics.

Abstract

Active Curriculum Language Modeling (ACLM; Hong et al., 2023) is a learner directed approach to training a language model. We proposed the original version of this process in our submission to the BabyLM 2023 task, and now we propose an updated ACLM process for the BabyLM 2024 task. ACLM involves an iteratively- and dynamically-constructed curriculum informed over the training process by a model of uncertainty; other training items that are similarly uncertain to a least certain candidate item are prioritized. Our new process improves the similarity model so that it is more dynamic, and we run ACLM over the most successful model from the BabyLM 2023 task: ELC-BERT (Charpentier and Samuel, 2023). We find that while our models underperform on fine-grained grammatical inferences, they outperform the BabyLM 2024 official base-lines on common-sense and world-knowledge tasks. We make our code available at https: //github.com/asayeed/ActiveBaby.

A surprisal oracle for when every layer counts

TL;DR

The paper addresses training language models under data-constrained BabyLM conditions by introducing a dynamic, learner-directed curriculum (ACLM) that recomputes a surprisal space at every training iteration using the current model (ELC-BERT). It evaluates ACLM on the BabyLM 2024 strict-small task, showing that while the approach does not consistently outperform the strongest baselines on fine-grained grammatical tasks, it yields improvements on world-knowledge (EWOK) and GLUE tasks, suggesting benefits from learning-order dynamics. The contributions include a dynamic surprisal-space update mechanism, 128-token sequence processing, and a kNN-based uncertainty-driven curriculum without human-in-the-loop, highlighting the cognitive plausibility of learner-directed training. The work indicates that ACLM can augment high-performing architectures in data-limited regimes and points to future work on hyperparameter exploration and alternative uncertainty signals to further unlock learner-like language acquisition dynamics.

Abstract

Active Curriculum Language Modeling (ACLM; Hong et al., 2023) is a learner directed approach to training a language model. We proposed the original version of this process in our submission to the BabyLM 2023 task, and now we propose an updated ACLM process for the BabyLM 2024 task. ACLM involves an iteratively- and dynamically-constructed curriculum informed over the training process by a model of uncertainty; other training items that are similarly uncertain to a least certain candidate item are prioritized. Our new process improves the similarity model so that it is more dynamic, and we run ACLM over the most successful model from the BabyLM 2023 task: ELC-BERT (Charpentier and Samuel, 2023). We find that while our models underperform on fine-grained grammatical inferences, they outperform the BabyLM 2024 official base-lines on common-sense and world-knowledge tasks. We make our code available at https: //github.com/asayeed/ActiveBaby.

Paper Structure

This paper contains 6 sections, 1 figure, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Learner-directed Active Curriculum Language Modeling
  4. Analysis
  5. Conclusions and future work
  6. Pre-training details

Figures (1)

  • Figure 1: The architecture of our ACLM method from last year's submission, described in hong-etal-2023-surprisal. For this study, we modify the trigram surpisal similarity model to simply use the average sentence surprisal of the model under training, which is now ELC-BERT rather than RoBERTa.