Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping
Minjia Zhang, Yuxiong He
TL;DR
This work tackles the heavy computational cost of unsupervised pre-training for Transformer-based language models by introducing Progressive Layer Dropping (PLD). It combines Switchable-Transformer blocks with identity-mapping (PreLN) and per-layer gates, guided by a progressive schedule that drops depth over training while keeping a floor keep-rate. Empirical results on BERT show 24% faster per-sample training and up to 2.5× faster pre-training to reach similar downstream accuracy, with GLUE performance matching or exceeding the baseline under equivalent data budgets. The approach yields strong knowledge transferability and robustness to hyperparameters, offering a practical route to faster pre-training without requiring massive hardware investments.
Abstract
Recently, Transformer-based language models have demonstrated remarkable performance across many NLP domains. However, the unsupervised pre-training step of these models suffers from unbearable overall computational expenses. Current methods for accelerating the pre-training either rely on massive parallelism with advanced hardware or are not applicable to language modeling. In this work, we propose a method based on progressive layer dropping that speeds the training of Transformer-based language models, not at the cost of excessive hardware resources but from model architecture change and training technique boosted efficiency. Extensive experiments on BERT show that the proposed method achieves a 24% time reduction on average per sample and allows the pre-training to be 2.5 times faster than the baseline to get a similar accuracy on downstream tasks. While being faster, our pre-trained models are equipped with strong knowledge transferability, achieving comparable and sometimes higher GLUE score than the baseline when pre-trained with the same number of samples.