On Importance of Layer Pruning for Smaller BERT Models and Low Resource Languages

TL;DR

The paper investigates layer pruning as a means to produce smaller, task-specific BERT variants for Marathi, a low-resource language. By evaluating MahaBERT-v2 and Google-Muril alongside compact scratch-trained baselines on SHC, LPC, and LDC datasets, it shows that middle-layer pruning often yields favorable size–accuracy trade-offs and that pruned models can match or surpass similarly sized scratch-trained counterparts. The findings emphasize that monolingual BERTs tend to outperform multilingual ones under these settings, and that pruning from the middle provides a practical, compute-efficient path to deployment without full pre-training. The work offers practical guidance for applying pruning to Indic NLP tasks and motivates extending the approach to additional low-resource languages.

Abstract

This study explores the effectiveness of layer pruning for developing more efficient BERT models tailored to specific downstream tasks in low-resource languages. Our primary objective is to evaluate whether pruned BERT models can maintain high performance while reducing model size and complexity. We experiment with several BERT variants, including MahaBERT-v2 and Google-Muril, applying different pruning strategies and comparing their performance to smaller, scratch-trained models like MahaBERT-Small and MahaBERT-Smaller. We fine-tune these models on Marathi datasets, specifically Short Headlines Classification (SHC), Long Paragraph Classification (LPC) and Long Document Classification (LDC), to assess their classification accuracy. Our findings demonstrate that pruned models, despite having fewer layers, achieve comparable performance to their fully-layered counterparts while consistently outperforming scratch-trained models of similar size. Notably, pruning layers from the middle of the model proves to be the most effective strategy, offering performance competitive with pruning from the top and bottom. However, there is no clear winner, as different pruning strategies perform better in different model and dataset combinations. Additionally, monolingual BERT models outperform multilingual ones in these experiments. This approach, which reduces computational demands, provides a faster and more efficient alternative to training smaller models from scratch, making advanced NLP models more accessible for low-resource languages without compromising classification accuracy.

Figures (1)

• Figure 1: Layer pruning strategies for BERT models.