From Local to Global: Revisiting Structured Pruning Paradigms for Large Language Models
Ziyan Wang, Enmao Diao, Qi Le, Pu Wang, Minwoo Lee, Shu-ping Yeh, Evgeny Stupachenko, Hao Feng, Li Yang
TL;DR
This work addresses the challenge of efficiently compressing and deploying large language models by moving beyond task-agnostic local pruning to a model-wide, global paradigm. It introduces GISP, a post-training Global Iterative Structured Pruning framework that computes structure-level importance using first-order loss information and applies a ratio-scheduled, iterative pruning process to select attention heads and MLP channels, yielding nested subnetworks suitable for a prune-once, deploy-many workflow. By redefining importance with respect to a task objective (e.g., perplexity for language modeling or a margin loss for decision-style tasks), GISP can be directly aligned with downstream goals, enabling task-specific pruning in a post-training setting. Empirical results across Llama2-7B/13B, Llama3-8B, Mistral-0.3-7B, and DeepSeek-R1-Distill-Llama-3-8B show consistent perplexity reductions and improved downstream accuracy, with strongest gains at 40–50% sparsity and notable gains for task-aligned calibration on CMQA and GSM8K. The approach supports a true once-for-all capability, offering an adaptable, efficient pathway for deploying sparsified LLMs without per-target retraining, while highlighting memory costs and future extensions to PEFT and MoE models.
Abstract
Structured pruning is a practical approach to deploying large language models (LLMs) efficiently, as it yields compact, hardware-friendly architectures. However, the dominant local paradigm is task-agnostic: by optimizing layer-wise reconstruction rather than task objectives, it tends to preserve perplexity or generic zero-shot behavior but fails to capitalize on modest task-specific calibration signals, often yielding limited downstream gains. We revisit global structured pruning and present GISP-Global Iterative Structured Pruning-a post-training method that removes attention heads and MLP channels using first-order, loss-based important weights aggregated at the structure level with block-wise normalization. An iterative schedule, rather than one-shot pruning, stabilizes accuracy at higher sparsity and mitigates perplexity collapse without requiring intermediate fine-tuning; the pruning trajectory also forms nested subnetworks that support a "prune-once, deploy-many" workflow. Furthermore, because importance is defined by a model-level loss, GISP naturally supports task-specific objectives; we instantiate perplexity for language modeling and a margin-based objective for decision-style tasks. Extensive experiments show that across Llama2-7B/13B, Llama3-8B, and Mistral-0.3-7B, GISP consistently lowers WikiText-2 perplexity and improves downstream accuracy, with especially strong gains at 40-50% sparsity; on DeepSeek-R1-Distill-Llama-3-8B with GSM8K, task-aligned calibration substantially boosts exact-match accuracy.