BabyLlama-2: Ensemble-Distilled Models Consistently Outperform Teachers With Limited Data

TL;DR

Through an extensive hyperparameter sweep, it is demonstrated that the advantages of distillation cannot be attributed to suboptimal hyperparameter selection of the teachers and underscore the need for further investigation into distillation techniques, particularly in data-limited settings.

Abstract

We present BabyLlama-2, a 345 million parameter model distillation-pretrained from two teachers on a 10 million word corpus for the BabyLM competition. On BLiMP and SuperGLUE benchmarks, BabyLlama-2 outperforms baselines trained on both 10 and 100 million word datasets with the same data mix, as well as its teacher models. Through an extensive hyperparameter sweep, we demonstrate that the advantages of distillation cannot be attributed to suboptimal hyperparameter selection of the teachers. Our findings underscore the need for further investigation into distillation techniques, particularly in data-limited settings.

Figures (3)

• Figure 1: Comparison of the models for each evaluation metric, in the form of violin plots, with ticks denoting the mean and $\pm 1$ standard deviation. The baselines are denoted by square and triangle markers, the submitted model (BabyLlama-2) by stars, and the best checkpoint from the entire hyperparameter sweep by a cross. BabyLlama (100M) and LTG-BERT(100M) were trained on the 100M dataset.
• Figure 2: BLiMP scores (averaged over all sub-tasks) as a function of the validation loss. Every circle represents a model from the hyperparameter sweep.
• Figure 3: Cross-entropy loss as a function of dataset size for 16M and 345M models.