Learning Dynamics of Meta-Learning in Small Model Pretraining

TL;DR

This work asks whether meta-learning can make pretraining of small language models faster and more interpretable. It integrates first-order MAML with subset-masked language modeling tasks to pretrain decoder-only models across four sizes (11M–570M). The study reports modest systematic gains on Universal NER, alongside a distinctive learning-dynamics signature: an initial diversification of representations followed by compression into a shared subspace, captured by effective-rank and head-entropy metrics. The authors release code, checkpoints, and logs, and discuss capacity-dependent effects, zero-shot transfers in low-resource languages, and phase-transition-like behavior that informs interpretability of meta-adaptation. Overall, the approach demonstrates potential for accelerating pretraining of small LMs and provides tools for diagnosing and understanding meta-learning dynamics, while outlining limitations and avenues for broader evaluation.

Abstract

Large language models are powerful but costly. We ask whether meta-learning can make the pretraining of small language models not only better but also more interpretable. We integrate first-order MAML with subset-masked LM pretraining, producing four LLama-style decoder-only models (11M-570M params), and evaluate it on a fundamental NLP task with many settings and real-world applications. Compared with vanilla training, our model (i) reaches the same loss up to 1.6x sooner, (ii) improves F1 on multilingual Universal NER under equal compute, and (iii) makes the training dynamics easy to read: first the network's representations fan out ("diversify") and later they collapse into a smaller, shared subspace ("compress"). This two-stage shift shows up as a rise-and-fall in both effective-rank curves and attention-head entropy. The same curves pinpoint which layers specialise earliest and which later reconverge, giving a compact, interpretable signature of meta-adaptation. Code, checkpoints and WandB logs are released.

Figures (6)

• Figure 1: Training loss and Paloma perplexity across pretraining steps for all MAML models. Two‐panel plot showing the evolution of (top) cross‐entropy training loss and (bottom) Paloma perplexity, each as a function of global pretraining step.
• Figure 2: MAML-Vanilla micro‐F1 difference by entity class and tuning regime, averaged across in-language datasets. Grouped bar charts reporting $\Delta$F1 = F1 MAML - F1 (Vanilla) for three named‐entity classes-PERSON (PER), LOCATION (LOC) and ORGANIZATION (ORG)-for pico-MAML decoders of four sizes (tiny, small, medium, large), averaged over nine in‐language NER datasets, over two fine-tuning regimes.
• Figure 3: Average support and query accuracy across pretraining steps for all models. Top: Average support‐set accuracy (%) measured at the end of each inner‐loop adaptation, as a function of the global pretraining step. Bottom: Corresponding average query‐set accuracy (%) after adaptation.
• Figure 4: Evolution of classifier head weights during meta-training. Top: Standard deviation of the task-specific classifier head weights (in logits space). Bottom: Mean of the classifier head weights.
• Figure 5: Proportional effective rank of MAML and vanilla models on available checkpoints until 6k steps. Top: weights; bottom: gradients.