BlockPruner: Fine-grained Pruning for Large Language Models

TL;DR

BlockPruner introduces a training-free, fine-grained pruning approach for LLMs by decomposing Transformer layers into minimal residual blocks (MHA and MLP) and using perplexity-based block importance within an iterative pruning framework. It demonstrates that block-level redundancy is common and more impactful than layer-level redundancy, enabling aggressive pruning with minimal performance loss and substantial inference speedups via reduced KV cache usage. Across multiple model families and benchmarks, BlockPruner outperforms state-of-the-art structured pruning baselines, especially at higher sparsity, and reveals that MHA blocks exhibit more redundancy than MLP blocks, particularly in larger models. The work also analyzes metric choices and data effects, showing perplexity-plus-iteration is superior to local similarity metrics and that Alpaca data yields better pruning guidance than WikiText-2. These findings advance practical LLM compression by enabling finer-grained pruning with favorable accuracy-speed trade-offs and broader generalization to new model families.

Abstract

With the rapid growth in the size and complexity of large language models (LLMs), the costs associated with their training and inference have escalated significantly. Research indicates that certain layers in LLMs harbor substantial redundancy, and pruning these layers has minimal impact on the overall performance. While various layer pruning methods have been developed based on this insight, they generally overlook the finer-grained redundancies within the layers themselves. In this paper, we delve deeper into the architecture of LLMs and demonstrate that finer-grained pruning can be achieved by targeting redundancies in multi-head attention (MHA) and multi-layer perceptron (MLP) blocks. We propose a novel, training-free structured pruning approach called BlockPruner. Unlike existing layer pruning methods, BlockPruner segments each Transformer layer into MHA and MLP blocks. It then assesses the importance of these blocks using perplexity measures and applies a heuristic search for iterative pruning. We applied BlockPruner to LLMs of various sizes and architectures and validated its performance across a wide range of downstream tasks. Experimental results show that BlockPruner achieves more granular and effective pruning compared to state-of-the-art baselines.

Figures (10)

• Figure 1: Block Influence (BI) scores men2024shortgpt for the Llama2-7B model touvron2023llama2 computed at both layer and block levels, where blocks/layers with lower BI scores indicate less importance. The model has 32 Transformer layers, each containing one MHA and one MLP block, totaling 64 blocks. Block-level BI scores are generally lower than layer-level scores, indicating finer-grained redundancies.
• Figure 2: Illustration depicting that a Transformer layer can be subdivided into two residual blocks.
• Figure 3: Overview of our BlockPruner. We iteratively calculate the importance score for each block (MHA or MLP) to obtain the block importance distribution, and subsequently remove the block with the lowest importance.
• Figure 4: The impact of pruning MHA and MLP individually with different pruning ratios on model performance. "MHA&MLP" represents the original BlockPruner algorithm. Results show that MHA modules in LLMs are more redundant than MLP modules.
• Figure 5: Left: The proportion of MHA blocks removed during the pruning process, relative to the total number of removed blocks. Right: The number of different blocks removed from models at a pruning ratio of 30%.