Lightweight and Post-Training Structured Pruning for On-Device Large Lanaguage Models

TL;DR

COMP tackles on-device pruning of large language models by introducing a hybrid-granularity, post-training pruning framework that combines layer-grained and neuron-grained pruning with mask tuning. It relies on a matrix-condition-based neuron importance metric and dynamically loads layers to minimize memory usage, enabling pruning with as little as minimal calibration data. Across LLaMA-2, OPT, and ChatGLM3 models, COMP outperforms state-of-the-art methods at 20% pruning (e.g., up to a 6.13 percentage point improvement) while substantially reducing memory overhead, demonstrating practical viability for edge devices. The approach offers broad applicability across architectures and addresses privacy concerns by avoiding fine-tuning on cloud data, making on-device deployment more feasible and scalable.

Abstract

Considering the hardware-friendly characteristics and broad applicability, structured pruning has emerged as an efficient solution to reduce the resource demands of large language models (LLMs) on resource-constrained devices. Traditional structured pruning methods often need fine-tuning to recover performance loss, which incurs high memory overhead and substantial data requirements, rendering them unsuitable for on-device applications. Additionally, post-training structured pruning techniques typically necessitate specific activation functions or architectural modifications, thereby limiting their scope of applications. Herein, we introduce COMP, a lightweight post-training structured pruning method that employs a hybrid-granularity pruning strategy. COMP initially prunes selected model layers based on their importance at a coarse granularity, followed by fine-grained neuron pruning within the dense layers of each remaining model layer. To more accurately evaluate neuron importance, COMP introduces a new matrix condition-based metric. Subsequently, COMP utilizes mask tuning to recover accuracy without the need for fine-tuning, significantly reducing memory consumption. Experimental results demonstrate that COMP improves performance by 6.13\% on the LLaMA-2-7B model with a 20\% pruning ratio compared to LLM-Pruner, while simultaneously reducing memory overhead by 80\%.

Figures (5)

• Figure 1: Layer redundancy of different models.
• Figure 2: Wikitext2 perplexity of different models with three strategies under certain pruning ratios.
• Figure 3: Illustration of the pruning process of COMP.
• Figure 4: GPU memory cost with different pruning methods.
• Figure 5: The perplexity variation with the numbers of removed layers in LLaMA-2-7B.