Can Model Compression Improve NLP Fairness

TL;DR

This paper investigates how model compression via Knowledge Distillation and Pruning affects toxicity and social bias in generative NLP models, using GPT-2 as the testbed. It reports a consistent decline in toxicity—and a general trend toward reduced bias—with distillation as model size decreases, supporting the memorization hypothesis and framing compression as a potential regularizer for fairness. The work extends the idea of compression as regularization to generative language models and discusses practical implications for safe deployment of compressed LMs. It also highlights limitations in bias benchmarks and proposes future work to deepen theoretical understanding and evaluation.

Abstract

Model compression techniques are receiving increasing attention; however, the effect of compression on model fairness is still under explored. This is the first paper to examine the effect of distillation and pruning on the toxicity and bias of generative language models. We test Knowledge Distillation and Pruning methods on the GPT2 model and found a consistent pattern of toxicity and bias reduction after model distillation; this result can be potentially interpreted by existing line of research which describes model compression as a regularization technique; our work not only serves as a reference for safe deployment of compressed models, but also extends the discussion of "compression as regularization" into the setting of neural LMs, and hints at the possibility of using compression to develop fairer models.

Figures (9)

• Figure 1: Different classes of distilled models
• Figure 2: Stereoset sample
• Figure 3: Perplexity of distilled models
• Figure 4: Perplexity of pruned models
• Figure 5: Toxicity tested on TCCC