Bias Dynamics in BabyLMs: Towards a Compute-Efficient Sandbox for Democratising Pre-Training Debiasing

TL;DR

This work tackles the high cost of debiasing large language models by introducing BabyLMs as compute-efficient proxies that replicate bias acquisition and debiasing dynamics. It demonstrates that BabyLMs exhibit bias–performance trajectories and debiasing responses closely aligned with standard LMs, enabling reliable pre-model debiasing studies at a fraction of the compute. Through a range of pre-model interventions (CDA, toxicity removal, perturbation augmentation) and analyses of corpora, the paper shows that toxicity and demographic imbalance are key drivers of bias, and that results generalize across seeds and architectures. The findings suggest a practical pathway to democratise debiasing research, accelerating exploration of fairer LMs while reducing resource requirements, albeit with careful validation on larger, real-world models.

Abstract

Pre-trained language models (LMs) have, over the last few years, grown substantially in both societal adoption and training costs. This rapid growth in size has constrained progress in understanding and mitigating their biases. Since re-training LMs is prohibitively expensive, most debiasing work has focused on post-hoc or masking-based strategies, which often fail to address the underlying causes of bias. In this work, we seek to democratise pre-model debiasing research by using low-cost proxy models. Specifically, we investigate BabyLMs, compact BERT-like models trained on small and mutable corpora that can approximate bias acquisition and learning dynamics of larger models. We show that BabyLMs display closely aligned patterns of intrinsic bias formation and performance development compared to standard BERT models, despite their drastically reduced size. Furthermore, correlations between BabyLMs and BERT hold across multiple intra-model and post-model debiasing methods. Leveraging these similarities, we conduct pre-model debiasing experiments with BabyLMs, replicating prior findings and presenting new insights regarding the influence of gender imbalance and toxicity on bias formation. Our results demonstrate that BabyLMs can serve as an effective sandbox for large-scale LMs, reducing pre-training costs from over 500 GPU-hours to under 30 GPU-hours. This provides a way to democratise pre-model debiasing research and enables faster, more accessible exploration of methods for building fairer LMs.

Figures (18)

• Figure 1: Bias and performance changes caused by post-model debiasing strategies
• Figure 2: Bias and performance changes caused by intra-model debiasing strategies
• Figure 3: Bias and performance metrics evaluation from pre-training on the CDA and CDA-ablation corpora
• Figure 4: Bias and performance variance during CDA pre-training across random seeds {1, 10, 50, 100}
• Figure 5: Evolution of bias and performance during pre-training under three corpus strategies: LLM detoxification, removing toxicity, and toxicity-removal ablation