Mask and You Shall Receive: Optimizing Masked Language Modeling For Pretraining BabyLMs

TL;DR

The paper tackles data- and time-efficient BabyLM pretraining by introducing curriculum-inspired Adaptive MLM (AMLM) and token-level N-hot embeddings to incorporate subtoken information. AMLM weighs masking at the token level using accuracy-based hard scores or loss-based soft scores and updates weights with a decaying rule while maintaining an average mask rate $p_{mlm}$. The authors report improved (Super)GLUE performance and stronger adjective nominalization signals, with ablations showing that hard masking and decaying schedules generally outperform standard MLM, while N-hot embeddings yield targeted gains on morphology tasks. Overall, the work demonstrates that curriculum-like masking strategies and finer-grained embeddings can boost efficiency and morpho-grammatical understanding in low-resource pretraining.

Abstract

We describe our strategy for the 2025 edition of the BabyLM Challenge. Our main contribution is that of an improved form of Masked Language Modeling (MLM), which adapts the probabilities of the tokens masked according to the model's ability to predict them. The results show a substantial increase in performance on (Super)GLUE tasks over the standard MLM. We also incorporate sub-token embeddings, finding that this increases the model's morphological generalization capabilities. Our submission beats the baseline in the strict-small track.

Figures (4)

• Figure 1: Token masking probabilities for hard method, grouped by frequency rank (in groups of 1000). Lower rank indicates higher frequency (e.g., blue is the most frequent group of words).
• Figure 2: Token masking probabilities for soft method, grouped by frequency rank (in groups of 1000). Lower rank indicates higher frequency.
• Figure 3: Masking probabilities by POS tag for hard method.
• Figure 4: Masking probabilities by POS tag for soft method.