Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score

Jimyung Hong, Jaehyung Kim

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities, but their massive scale poses significant challenges for practical deployment. Structured pruning offers a promising solution by removing entire dimensions or layers, yet existing methods face critical trade-offs: task-agnostic approaches cannot adapt to task-specific requirements, while task-aware methods require costly training to learn task adaptability. We propose DIET (Dimension-wise global pruning of LLMs via merging Task-wise importance scores), a training-free structured pruning method that combines dimension-level granularity with task-aware selection. DIET profiles activation magnitudes across tasks using only 100 samples per task, then applies majority voting to construct a single global mask. DIET does not require large costs from pre-computation or training. Experiments on seven zero-shot benchmarks using Gemma-2 2B and 9B models demonstrate the effectiveness of DIET; for example, at 20% sparsity on Gemma-2 2B, DIET achieves near 10% average accuracy improvement, compared to previous state-of-the-art structured pruning methods. This advantage persists across various sparsity levels and model scales, positioning DIET as a practical and robust choice for structured LLM pruning.

Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities, but their massive scale poses significant challenges for practical deployment. Structured pruning offers a promising solution by removing entire dimensions or layers, yet existing methods face critical trade-offs: task-agnostic approaches cannot adapt to task-specific requirements, while task-aware methods require costly training to learn task adaptability. We propose DIET (Dimension-wise global pruning of LLMs via merging Task-wise importance scores), a training-free structured pruning method that combines dimension-level granularity with task-aware selection. DIET profiles activation magnitudes across tasks using only 100 samples per task, then applies majority voting to construct a single global mask. DIET does not require large costs from pre-computation or training. Experiments on seven zero-shot benchmarks using Gemma-2 2B and 9B models demonstrate the effectiveness of DIET; for example, at 20% sparsity on Gemma-2 2B, DIET achieves near 10% average accuracy improvement, compared to previous state-of-the-art structured pruning methods. This advantage persists across various sparsity levels and model scales, positioning DIET as a practical and robust choice for structured LLM pruning.
Paper Structure (45 sections, 8 equations, 3 figures, 23 tables)

This paper contains 45 sections, 8 equations, 3 figures, 23 tables.

Table of Contents

  1. Introduction
  2. Dimension-wise Global Pruning via Merging Task-wise Importance Scores
  3. Preliminaries
  4. Mathematical formulation.
  5. DieT: Merge Task-wise Importance for Global Dimension-wise Pruning of LLMs
  6. Task-wise importance scoring.
  7. Merging into global pruning mask.
  8. From Global Masking to Hard Pruning for Real-Time Efficiency
  9. Variance correction via weight rescaling.
  10. Experiments
  11. Setups
  12. Datasets.
  13. Metrics.
  14. Baselines.
  15. Implementation details.
  16. ...and 30 more sections

Figures (3)

  • Figure 1: Step 1. Task-wise activation profiling is performed on a dense LLM with hidden width $d$. Step 2. A global pruning mask is constructed via task-aware aggregation: for each task $t$, a binary selector selects low-importance dimensions based on activation scores. These task-wise votes are aggregated through majority voting to produce the global mask $m \in \{0,1\}^d$. Step 3. The mask is applied across all layers, reducing the residual width from $d$ to $d'$.
  • Figure 2: Consensus-vote histogram by dimension across 7 tasks on Gemma-2 2B. Numbers show total counts (and share of $d{=}2304$).
  • Figure 3: Zero-shot accuracy by activation-profiling sample size on Gemma-2 2B across seven benchmarks. Benchmarks with * use acc_norm.