CLASS-IT: Conversational and Lecture-Aligned Small-Scale Instruction Tuning for BabyLMs

TL;DR

This study investigates whether BabyLM-scale, ecologically realistic LMs benefit from instruction tuning by contrasting conversational and QA instruction data delivered in merged versus sequential curricula, using decoder-only models of ~140M and ~100M parameters. Evaluations span both fine-tuning on SuperGLUE and zero-shot tasks (BLiMP, EWoK, WUGs, Entity Tracking, and psycholinguistic correlations), revealing modest gains in fine-tuning—especially with sequential curricula—while zero-shot transfer remains inconsistent, indicating a trade-off between interaction-oriented adaptation and broad linguistic generalization. The findings illustrate both potential and limits of applying human-inspired learning strategies to small LMs and advocate for hybrid, curriculum-based approaches to bolster generalization under strict data and compute constraints. Limitations include reliance on Switchboard-style conversational data and evaluation criteria; future work should explore richer interactive corpora and multi-task curricula to better balance interaction and generalization.

Abstract

This work investigates whether small-scale LMs can benefit from instruction tuning. We compare conversational and question-answering instruction tuning datasets, applied either in a merged or sequential curriculum, using decoder-only models with 100M and 140M parameters. Evaluation spans both fine-tuning (SuperGLUE) and zero-shot (BLiMP, EWoK, WUGs, entity tracking, and psycholinguistic correlation) settings. Results show that instruction tuning yields small but consistent gains in fine-tuning scenarios, with sequential curricula outperforming merged data; however, improvements do not consistently transfer to zero-shot tasks, suggesting a trade-off between interaction-focused adaptation and broad linguistic generalization. These results highlight both the potential and the constraints of adapting human-inspired learning strategies to low-resource LMs, and point toward hybrid, curriculum-based approaches for enhancing generalization under ecological training limits.

Figures (6)

• Figure 1:
• Figure 2: Results of fine-tuned models on (Super)Glue tasks.
• Figure 3: Median, Inter-Quartile Ranges (IQR), and outliers for z-scores of each model in the fine-tuning evaluation.
• Figure 4: Results of the zero-shot evaluation. Tasks measured with accuracy are reported in the left bar chart; tasks measured with change in $R^2$ are reported in the bar chart on the right.
• Figure 5: Median, Inter-Quartile Ranges (IQR), and outliers for z-scores of each model in the zero-shot evaluation.