Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning

Minsik Choi, Hyegang Son, Changhoon Kim, Young Geun Kim

TL;DR

Transformer pruning is challenged by gradient-based HIS alone, which ignores per-head attention distribution and can destabilize pruning. The paper introduces Head Importance Entropy Score (HIES), defined as $\mathrm{HIES}_h = \alpha \widehat{\mathrm{HIS}}_h + (1-\alpha)(1-\widehat{\mathrm{AE}}_h)$ with min–max normalization to $[0,1]$, enabling layer-adaptive pruning. The authors prove near-orthogonality of HIS and AE gradients and demonstrate that HIES yields more stable and accurate pruning than HIS alone, with up to 15.2% accuracy gains and 2.04x stability improvements on aggressive pruning. Their experiments scale to larger transformers and multimodal models, indicating broad applicability for resource-constrained deployment. Code release is planned.

Abstract

Transformer-based models have achieved remarkable performance in NLP tasks. However, their structural characteristics-multiple layers and attention heads-introduce efficiency challenges in inference and deployment. To address these challenges, various pruning methods have recently been proposed. Notably, gradient-based methods using Head Importance Scores (HIS) have gained traction for interpretability, efficiency, and ability to identify redundant heads. However, HIS alone has limitations as it captures only the gradient-driven contribution, overlooking the diversity of attention patterns. To overcome these limitations, we introduce a novel pruning criterion, HIES (Head Importance-Entropy Score), which integrates head importance scores with attention entropy, providing complementary evidence on per-head contribution. Empirically, HIES-based pruning yields up to 15.2% improvement in model quality and 2.04x improvement in stability over HIS-only methods, enabling substantial model compression without sacrificing either accuracy or stability. Code will be released upon publication.

Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning

TL;DR

Transformer pruning is challenged by gradient-based HIS alone, which ignores per-head attention distribution and can destabilize pruning. The paper introduces Head Importance Entropy Score (HIES), defined as with min–max normalization to , enabling layer-adaptive pruning. The authors prove near-orthogonality of HIS and AE gradients and demonstrate that HIES yields more stable and accurate pruning than HIS alone, with up to 15.2% accuracy gains and 2.04x stability improvements on aggressive pruning. Their experiments scale to larger transformers and multimodal models, indicating broad applicability for resource-constrained deployment. Code release is planned.

Abstract

Transformer-based models have achieved remarkable performance in NLP tasks. However, their structural characteristics-multiple layers and attention heads-introduce efficiency challenges in inference and deployment. To address these challenges, various pruning methods have recently been proposed. Notably, gradient-based methods using Head Importance Scores (HIS) have gained traction for interpretability, efficiency, and ability to identify redundant heads. However, HIS alone has limitations as it captures only the gradient-driven contribution, overlooking the diversity of attention patterns. To overcome these limitations, we introduce a novel pruning criterion, HIES (Head Importance-Entropy Score), which integrates head importance scores with attention entropy, providing complementary evidence on per-head contribution. Empirically, HIES-based pruning yields up to 15.2% improvement in model quality and 2.04x improvement in stability over HIS-only methods, enabling substantial model compression without sacrificing either accuracy or stability. Code will be released upon publication.
Paper Structure (73 sections, 4 theorems, 49 equations, 28 figures, 6 tables)

This paper contains 73 sections, 4 theorems, 49 equations, 28 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Motivation
  4. Proposed Method
  5. Head Importance Entropy Score
  6. Theoretical Analysis
  7. Loss-Increase Control via Head Importance (HIS)
  8. Setup.
  9. Head Importance Score (HIS).
  10. Implication for pruning.
  11. Generalization Gap and Attention Entropy (AE)
  12. Setup.
  13. Notation (attention entropy deficit).
  14. Main bound (loss--entropy link).
  15. Interpretation.
  16. ...and 58 more sections

Key Result

Lemma 1

Let $\beta_y:=\|\nabla_{\mathbf y}^2\mathcal{L}\|_2$. For any mask variables $\{m_h\}_{h=1}^{|\mathrm{H}|}$, $\Delta\mathcal{L} ~:=~ \mathbb E_{\mathcal{D}}\!\bigl[\mathcal{L}(\mathbf y+\delta\mathbf y)-\mathcal{L}(\mathbf y)\bigr] ~\le~ \sum_{h=1}^{|\mathrm{H}|} (1-m_h)\,\mathrm{HIS}_h ~+~ \frac{\b

Figures (28)

  • Figure 1: Analysis of accuracy degradation and head behaviors under HIS-based pruning. In our diagnostic study, we analyze the phenomena of pruning by HIS on BERT, focusing on detailed attention head behaviors during inference. (a) Accuracy curves of HIS-based pruning on CoLA and SST-2. The bold color highlights the sharp-drop regime of HIS-based pruning. (b) Head-level analysis on SST-2 with a validation example misclassified by the HIS-pruned model. The attention score heatmap shows heads on each column, where pruned low-HIS heads are indicated with colored layer–head labels and unpruned high-HIS heads with gray table. It then shows token-wise distributions, where tokens deemed important for classification (e.g., sentiment-discriminative tokens in SST-2) are marked with O, and non-critical tokens with X. The left plot shows the distribution of heads by normalized HIS, and the right plot shows the distribution by normalized AE, where pruned low-HIS heads are highlighted with red boxes and unpruned high-HIS heads with gray boxes. See Appendix \ref{['Experimental Setups']} for experimental setup details.
  • Figure 2: Design overview of Head Importance-Entropy Score (HIES). Darker cells correspond to values closer to 1, while lighter cells correspond to values closer to 0. Note the heatmap utilizes each metric across attention heads in BERT on the CoLA dataset.
  • Figure 3: Head-level pruning patterns on the CoLA dataset across pruning ratios from 30% to 50% pruning ratios. Pruned heads are shaded in gray.
  • Figure 4: Comparison of HIES (Ours) and HIS on HellaSwag, Winogrande, ARC-e, ARC-c and OBQA across pruning ratios from 10% to 60% (x-axis). The top row reports task accuracy, while the bottom row reports stability. The rightmost panels summarize the mean over tasks.
  • Figure 5: Examples from VizWiz-VQA showing visual questions asked by blind users and the corresponding answers from crowd workers. The examples include both questions that can be answered from the image and questions that cannot.
  • ...and 23 more figures

Theorems & Definitions (7)

  • Lemma 1: Loss-increase upper bound under head masking
  • Lemma 2: Optimality
  • Lemma 3: Orthogonality
  • Lemma 4: Entropy–TV inequality
  • proof
  • proof
  • proof