What Happens When Small Is Made Smaller? Exploring the Impact of Compression on Small Data Pretrained Language Models

TL;DR

The paper investigates how well-established compression techniques—$distillation$, $pruning$, and $quantization$—translate to a low-resource, small-data setting using AfriBERTa. It evaluates these methods on AfriBERTa-base/large with MasakhaNER for NER, showing that distillation yields $22\%$–$33\%$ compression with competitive performance, pruning can reach around $60\%$ size reduction with modest losses, and quantization achieves about $64.08\%$ size reduction with roughly a $4.7\%$ average drop in F1 and significant speedups (up to $52.3\%$). Results also reveal language-specific trade-offs and potential cross-lingual benefits, including some languages surpassing dense baselines under compression. The findings demonstrate that compression techniques remain effective for small-data multilingual models, enabling deployment on resource-constrained devices, while highlighting the need to tailor strategies to language characteristics and task requirements.

Abstract

Compression techniques have been crucial in advancing machine learning by enabling efficient training and deployment of large-scale language models. However, these techniques have received limited attention in the context of low-resource language models, which are trained on even smaller amounts of data and under computational constraints, a scenario known as the "low-resource double-bind." This paper investigates the effectiveness of pruning, knowledge distillation, and quantization on an exclusively low-resourced, small-data language model, AfriBERTa. Through a battery of experiments, we assess the effects of compression on performance across several metrics beyond accuracy. Our study provides evidence that compression techniques significantly improve the efficiency and effectiveness of small-data language models, confirming that the prevailing beliefs regarding the effects of compression on large, heavily parameterized models hold true for less-parameterized, small-data models.

Figures (7)

• Figure 1: Pruning before vs after fine-tuning: F1 score of sparsified models, averaged across languages.
• Figure 2: A chart of the F1 scores for the best-performing student models and the teacher models across each language in the NER task.
• Figure 3: Performance (F1 score) during task agnostic and task-specific distillation across different languages in the NER task.
• Figure 4: Performance comparison between students and teachers (distillation).
• Figure 5: Pruning before vs after fine-tuning: F1 scores averaged across the performances of each language.