Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Data-Free Pruning of Self-Attention Layers in LLMs

Dhananjay Saikumar, Blesson Varghese

TL;DR

This work identifies Attention Suppression in deep self-attention layers of decoder-only Transformers, where later layers mute their own updates, transferring representational change to the residual path and MLP. Building on this, it introduces Gate-Norm, a data-free, one-shot pruning proxy that ranks attention layers purely from weight matrices via the gate norm $m_ ext{l} = \|W_{q, ext{l}} W_{k, ext{l}}^ op\|_F$, enabling fast, on-device pruning without calibration data or forward passes. Empirically, Gate-Norm matches data-driven pruning in perplexity and zero-shot accuracy across multiple 13B/7B checkpoints, while delivering up to 1.3–1.5x throughput gains and up to 30% speedups at modest pruning budgets; at larger budgets, gains persist with small accuracy degradations. The method runs in milliseconds on GPUs and under 30 seconds on CPUs, making data-free compression feasible for ultra-large models and privacy-constrained deployments, and is compatible with lightweight post-pruning LoRA fine-tuning. These findings suggest a path toward architectural simplifications and dynamic depth strategies that exploit inherent redundancy in late self-attention layers.

Abstract

Many self-attention sublayers in large language models (LLMs) can be removed with little to no loss. We attribute this to the Attention Suppression Hypothesis: during pre-training, some deep attention layers learn to mute their own contribution, leaving the residual stream and the MLP to carry the representation. We propose Gate-Norm, a one-shot, weight-only criterion that ranks attention sublayers by query--key coupling and removes the least coupled ones, requiring no calibration data, no forward passes, no fine-tuning, and no specialized kernels. On 40-layer, 13B-parameter LLaMA models, Gate-Norm prunes the model in under a second. Pruning $8$--$16$ attention sublayers yields up to $1.30\times$ higher inference throughput while keeping average zero-shot accuracy within $2\%$ of the unpruned baseline across BoolQ, RTE, HellaSwag, WinoGrande, ARC-Easy/Challenge, and OpenBookQA. Across these settings, Gate-Norm matches data-driven pruning methods in accuracy while being $\sim 1000\times$ faster to score layers, enabling practical, data-free compression of LLMs.

Data-Free Pruning of Self-Attention Layers in LLMs

TL;DR

This work identifies Attention Suppression in deep self-attention layers of decoder-only Transformers, where later layers mute their own updates, transferring representational change to the residual path and MLP. Building on this, it introduces Gate-Norm, a data-free, one-shot pruning proxy that ranks attention layers purely from weight matrices via the gate norm , enabling fast, on-device pruning without calibration data or forward passes. Empirically, Gate-Norm matches data-driven pruning in perplexity and zero-shot accuracy across multiple 13B/7B checkpoints, while delivering up to 1.3–1.5x throughput gains and up to 30% speedups at modest pruning budgets; at larger budgets, gains persist with small accuracy degradations. The method runs in milliseconds on GPUs and under 30 seconds on CPUs, making data-free compression feasible for ultra-large models and privacy-constrained deployments, and is compatible with lightweight post-pruning LoRA fine-tuning. These findings suggest a path toward architectural simplifications and dynamic depth strategies that exploit inherent redundancy in late self-attention layers.

Abstract

Many self-attention sublayers in large language models (LLMs) can be removed with little to no loss. We attribute this to the Attention Suppression Hypothesis: during pre-training, some deep attention layers learn to mute their own contribution, leaving the residual stream and the MLP to carry the representation. We propose Gate-Norm, a one-shot, weight-only criterion that ranks attention sublayers by query--key coupling and removes the least coupled ones, requiring no calibration data, no forward passes, no fine-tuning, and no specialized kernels. On 40-layer, 13B-parameter LLaMA models, Gate-Norm prunes the model in under a second. Pruning -- attention sublayers yields up to higher inference throughput while keeping average zero-shot accuracy within of the unpruned baseline across BoolQ, RTE, HellaSwag, WinoGrande, ARC-Easy/Challenge, and OpenBookQA. Across these settings, Gate-Norm matches data-driven pruning methods in accuracy while being faster to score layers, enabling practical, data-free compression of LLMs.
Paper Structure (39 sections, 41 equations, 5 figures, 3 tables, 1 algorithm)

This paper contains 39 sections, 41 equations, 5 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries and Notation
  4. Self-attention layer:
  5. Feed-forward (MLP) layer:
  6. Cosine‐Similarity Importance
  7. ShortGPT:
  8. Layer‐Level Attention vs. MLP Pruning
  9. Not All Attention Is Needed:
  10. Quantization and Unstructured Weight Pruning
  11. Attention Suppression Hypothesis
  12. Empirical Observation of Attention Suppression
  13. Cosine‐Similarity Importance Goes to Zero:
  14. From Cosine to Zero Update:
  15. Attention‐to‐Input Norm Ratio
  16. ...and 24 more sections

Figures (5)

  • Figure 1: Inference time of different layer types vs sequence length for LLaMA-7B.
  • Figure 2: Visualising attention drop. Removing the self-attention branch leaves a direct residual path feeding the MLP.
  • Figure 3: Attention‐to‐input norm ratio $r_\ell$ across layers 1–40 in the 40-layer LLaMA-13B. Early layers exhibit high ratios, mid layers plateau around 0.3, and deeper layers collapse toward zero, confirming that later attention updates become negligible.
  • Figure 4: WikiText-2 Perplexity vs. number of dropped Attention layers on LLaMA-13B v1 (left) and v2 (right), up to 25 layers. Top row: full-range curves (including Random). Bottom row: zoomed-in comparison (Data-driven vs Gate-Norm).
  • Figure 5: Pruned attention sub-layers in the 40-layer LLaMA-13B models. Columns compare model variants (v1 (left) vs. v2 (right)); rows compare pruning strategies (data-driven, top; data-free / Gate-Norm, bottom). Vertical bars indicate the attention sub-layers removed by each strategy.