Extracting General-use Transformers for Low-resource Languages via Knowledge Distillation

TL;DR

Low-resource languages suffer from limited pretraining data and expensive multilingual models. This work shows that simple knowledge distillation from a large MMT (mBERT) to a smaller language-specific transformer can yield robust, efficient single-language models. The distillation uses a combined objective $L_{distil} = \alpha_{KL} KL(out_{student} || out_{teacher}) + \alpha_{MLM} L_{MLM}(out_{student}, out_{teacher})$ with a temperature, producing dBERT Base and dBERT Tiny that perform well on hate-speech and NLI benchmarks and offer substantial training speedups. The results suggest a practical path for deploying high-quality, language-specific NLP models in low-resource settings, with clear ablations guiding future refinements and extensions to unseen languages and larger multilingual LLMs.

Abstract

In this paper, we propose the use of simple knowledge distillation to produce smaller and more efficient single-language transformers from Massively Multilingual Transformers (MMTs) to alleviate tradeoffs associated with the use of such in low-resource settings. Using Tagalog as a case study, we show that these smaller single-language models perform on-par with strong baselines in a variety of benchmark tasks in a much more efficient manner. Furthermore, we investigate additional steps during the distillation process that improves the soft-supervision of the target language, and provide a number of analyses and ablations to show the efficacy of the proposed method.