Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment

TL;DR

The paper tackles the limitation of single-granularity structured pruning for large language models by introducing HyWIA, an adaptive framework that fusion-labels weight importance across fine- and coarse-grained granularities using an attention mechanism. HyWIA comprises a grouping step to build dependency-based parameter groups, a hybrid-grained assessment that combines gradient signals from both granularities, and an adaptive fusion algorithm implemented as anAttention Fusion Model that dynamically weights the two signals without additional training. A LoRA-based fine-tuning step restores performance after pruning. Extensive experiments across LLaMA, Vicuna, Baichuan, and Bloom show HyWIA outperforms state-of-the-art pruning methods, with a notable average accuracy gain of 2.82% over LLM-Pruner at 50% pruning on LLaMA-7B, demonstrating improved accuracy and more uniform sparsity distribution with reduced hardware costs.

Abstract

Structured pruning for large language models (LLMs) has garnered significant academic interest due to its ability to efficiently compress and accelerate LLMs by eliminating redundant weight groups at a coarse-grained granularity. Current structured pruning methods for LLMs typically depend on a singular granularity for assessing weight importance, resulting in notable performance degradation in downstream tasks. Intriguingly, our empirical investigations reveal that utilizing unstructured pruning, which achieves better performance retention by pruning weights at a finer granularity, \emph{i.e.}, individual weights, yields significantly varied sparse LLM structures when juxtaposed to structured pruning. This suggests that evaluating both holistic and individual assessment for weight importance is essential for LLM pruning. Building on this insight, we introduce the Hybrid-grained Weight Importance Assessment (HyWIA), a novel method that merges fine-grained and coarse-grained evaluations of weight importance for the pruning of LLMs. Leveraging an attention mechanism, HyWIA adaptively determines the optimal blend of granularity in weight importance assessments in an end-to-end pruning manner. Extensive experiments on LLaMA-V1/V2, Vicuna, Baichuan, and Bloom across various benchmarks demonstrate the effectiveness of HyWIA in pruning LLMs. For example, HyWIA surpasses the cutting-edge LLM-Pruner by an average margin of 2.82% in accuracy across seven downstream tasks when pruning LLaMA-7B by 50%.

Figures (6)

• Figure 1: Sparsity allocation across different layers of LLaMA-7B pruned by fine-grained xia2022structured and coarse-grained lee2020flexible weight importance criteria (50% global pruning rate). Fine-grained pruning tends to preserve more weight in the shallow layers, which is in stark contrast to coarse-grained pruning. The vertical axis represents the parameter quantity of each layer in terms of millions. The horizontal axis represents the layer number of LLaMA-7B.
• Figure 2: The framework of our proposed Hybrid-grained Weight Importance Assessment (HyWIA) consists of three stages: grouping (blue), adaptive estimation (green), and fine-tuning (white). In the grouping stage, we construct the dependency structure within the LLM. The adaptive estimation stage includes gradient calculation, fine-grained and coarse-grained importance estimation, adaptive fusion, element sorting, and pruning. Finally, the fine-tuning stage uses LoRA hu2022lora to recover the pruned model's performance and functionality.
• Figure 3: On the left, the adaptive fusion rate is shown, where Out DG 1 i:0 (idx:4096) indicates the output channel (Out), direct connection group 1 (DG 1), the 0th sub-group (i:0), and 4096 parameters (idx:4096). For clarity, only the fusion rates of the first six parameters in the first three groups are displayed. On the right, adaptive pruning is compared with fine-grained and coarse-grained methods.
• Figure 4: Comparison of LLaMA-7B layer parameters before and after 50% pruning using our method.
• Figure 5: A line plot compares LLaMA-7B with adaptive pruning against fine-grained and coarse-grained methods, all with a 50% pruning rate.