Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Are Compressed Language Models Less Subgroup Robust?

Leonidas Gee, Andrea Zugarini, Novi Quadrianto

TL;DR

It is shown that worst-group performance does not depend on model size alone, but also on the compression method used, and that model compression does not always worsen the performance on minority subgroups.

Abstract

To reduce the inference cost of large language models, model compression is increasingly used to create smaller scalable models. However, little is known about their robustness to minority subgroups defined by the labels and attributes of a dataset. In this paper, we investigate the effects of 18 different compression methods and settings on the subgroup robustness of BERT language models. We show that worst-group performance does not depend on model size alone, but also on the compression method used. Additionally, we find that model compression does not always worsen the performance on minority subgroups. Altogether, our analysis serves to further research into the subgroup robustness of model compression.

Are Compressed Language Models Less Subgroup Robust?

TL;DR

It is shown that worst-group performance does not depend on model size alone, but also on the compression method used, and that model compression does not always worsen the performance on minority subgroups.

Abstract

To reduce the inference cost of large language models, model compression is increasingly used to create smaller scalable models. However, little is known about their robustness to minority subgroups defined by the labels and attributes of a dataset. In this paper, we investigate the effects of 18 different compression methods and settings on the subgroup robustness of BERT language models. We show that worst-group performance does not depend on model size alone, but also on the compression method used. Additionally, we find that model compression does not always worsen the performance on minority subgroups. Altogether, our analysis serves to further research into the subgroup robustness of model compression.
Paper Structure (24 sections, 3 figures, 5 tables)

This paper contains 24 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Experiments
  4. Models
  5. Knowledge Distillation (KD).
  6. Pruning.
  7. Quantization.
  8. Vocabulary Transfer (VT).
  9. Datasets
  10. MultiNLI.
  11. CivilComments.
  12. SCOTUS.
  13. Implementation Details
  14. Results
  15. Model Size and Subgroup Robustness.
  16. ...and 9 more sections

Figures (3)

  • Figure 1: Plot of WGA against average accuracy. Compression method is represented by marker type, while model size is represented by marker size. In MultiNLI and SCOTUS, compression worsens WGA for most models. Conversely, WGA improves for most compressed models in CivilComments.
  • Figure 2: Model performance is shown to improve across the binary datasets of MultiNLI. However, the overall trend in WGA remains relatively unchanged, with a decreasing model size leading to drops in WGA.
  • Figure 3: Distribution of accuracies by subgroup for KD. Sample sizes in the training set are shown beside each subgroup. In CivilComments, performance improves on minority subgroups (2 and 3) across most models as model size decreases contrary to the minority subgroups (3 and 5) of MultiNLI.