BabyLM Challenge: Exploring the Effect of Variation Sets on Language Model Training Efficiency

TL;DR

This work investigates how Variation Sets, a CDS-derived pattern of consecutive utterances with the same intent but varying form, affect language model training efficiency. By generating artificial VSs and mixing them with CDS at multiple ratios, the authors train a GPT-2 model and evaluate on BabyLM benchmarks BLiMP, EWOK, and GLUE, revealing that the optimal VS proportion is benchmark dependent and sensitive to how VSs are presented. The results show that VSs can improve certain linguistic competencies but that effects are nuanced by epoch count and data presentation order, with shuffled VSs sometimes outperforming sequential exposure. Overall, the study provides initial evidence that CDS-inspired VS patterns can enhance data efficiency in LM training, while highlighting substantial avenues for further methodological refinement and investigation.

Abstract

While current large language models have achieved a remarkable success, their data efficiency remains a challenge to overcome. Recently it has been suggested that child-directed speech (CDS) can improve training data efficiency of modern language models based on Transformer neural networks. However, it is not yet understood which specific properties of CDS are effective for training these models. In the context of the BabyLM Challenge, we focus on Variation Sets (VSs), sets of consecutive utterances expressing a similar intent with slightly different words and structures, which are ubiquitous in CDS. To assess the impact of VSs on training data efficiency, we augment CDS data with different proportions of artificial VSs and use these datasets to train an auto-regressive model, GPT-2. We find that the best proportion of VSs depends on the evaluation benchmark: BLiMP and GLUE scores benefit from the presence of VSs, but EWOK scores do not. Additionally, the results vary depending on multiple factors such as the number of epochs and the order of utterance presentation. Taken together, these findings suggest that VSs can have a beneficial influence on language models, while leaving room for further investigation.

Figures (1)

• Figure 1: Two methods for inputting VSs to the model during training. Each figure illustrates an example with a batch size of 3. The figure on the left shows the method of concatenating VSs into a single sequence. In this setting, the model always processes the sentences within a VS sequentially. The figure on the right shows the method of distributing each sentence of a VS into adjacent batches. In this setting, the model updates its parameters after observing each sentence in the VS before proceeding to the next sentence in the same set. In the figures, sentence $i$-$j$ indicates the $j$-th sentence in the $i$-th VS.