Efficient Shapley Value-based Non-Uniform Pruning of Large Language Models

TL;DR

The paper tackles efficient pruning of large language models by introducing SV-NUP, a Shapley-value-based non-uniform pruning framework that allocates pruning budgets by per-layer importance. To make Shapley-based budgeting scalable, it introduces Sliding Window SV (SWSV) to approximate layer contributions using local neighborhoods, reducing complexity from 2^T to O(T 2^{N-1}). Empirically, SV-NUP yields notable perplexity and zero-shot performance improvements over uniform pruning baselines (e.g., up to 18–19% PPL reduction on large LLaMA models at 70% sparsity) and shows robust performance across LLaMA and OPT families when integrated with methods like Magnitude, Wanda, and SparseGPT. The approach provides a theoretically grounded, practical path to non-uniform sparsity in LLMs, with potential extensions to joint pruning-quantization for further efficiency gains.

Abstract

Pruning large language models (LLMs) is a promising solution for reducing model sizes and computational complexity while preserving performance. Traditional layer-wise pruning methods often adopt a uniform sparsity approach across all layers, which leads to suboptimal performance due to the varying significance of individual transformer layers within the model not being accounted for. To this end, we propose the Shapley Value-based Non-Uniform Pruning (SV-NUP) method for LLMs. This approach quantifies the contribution of each transformer layer to the overall model performance, enabling the assignment of tailored pruning budgets to different layers to retain critical parameters. To further improve efficiency, we design the Sliding Window-based Shapley Value approximation method. It substantially reduces computational overhead compared to exact SV calculation methods. Extensive experiments on various LLMs including LLaMA-v1, LLaMA-v2 and OPT demonstrate the effectiveness of the proposed approach. The results reveal that non-uniform pruning significantly enhances the performance of pruned models. Notably, SV-NUP achieves a reduction in perplexity (PPL) of 18.01% and 19.55% on LLaMA-7B and LLaMA-13B, respectively, compared to SparseGPT at 70% sparsity.

Figures (9)

• Figure 1: Overview and effectiveness of SV-NUP: (a) Distribution of weight magnitude across Transformer layers; (b) Conceptual comparison of uniform pruning vs. SV-NUP; (c) PPL↓ (the smaller the better) comparison under different pruning strategies. The red number is the improvement percentage. Unlike uniform pruning, which ignores layer-wise importance and may degrade LLM performance, SV-NUP leverages Shapley value to estimate the contribution of each Transformer layer and allocates pruning ratios accordingly to better preserve model performance.
• Figure 2: Comparison of the mean and standard deviation (Std) across different Transformer layers under Magnitude. TinyLLaMA consists of 22 Transformer layers, LLama-7B and Mistral-7B consist of the same 32 Transformer layers, where each Transformer layer consists of 7 inner-layers.
• Figure 3: Contribution analysis of 3 Transformer layers in an LLM. For intuitive presentation, we use accuracy to represent the contribution of different Transformer layers.
• Figure 4: Selection of Transformer coalitions under an LLM with 8 Transformer layers.
• Figure 5: Illustration of SWSV.