IntraSlice: Towards High-Performance Structural Pruning with Block-Intra PCA for LLMs

TL;DR

IntraSlice tackles the deployment bottleneck of large language models by introducing block-intra PCA pruning within Transformer modules, avoiding activation-distribution disruption common to inter-module PCA methods. The framework combines adaptive head PCA, progressive sliced iterative PCA for nonlinear FFNs, and a PCA-based global non-uniform pruning estimator that accounts for compressed activation distributions, with transformation matrices fully fusionable into model weights. Empirical results on LLaMA2, LLaMA3, and Phi-series show superior compression performance at comparable or better inference speed than recent baselines, while maintaining or improving zero-shot task accuracy and perplexity. This approach enables high-performance, hardware-friendly pruning of LLMs, supporting more practical deployment in resource-constrained settings.

Abstract

Large Language Models (LLMs) achieve strong performance across diverse tasks but face deployment challenges due to their massive size. Structured pruning offers acceleration benefits but leads to significant performance degradation. Recent PCA-based pruning methods have alleviated this issue by retaining key activation components, but are only applied between modules in order to fuse the transformation matrix, which introduces extra parameters and severely disrupts activation distributions due to residual connections. To address these issues, we propose IntraSlice, a framework that applies block-wise module-intra PCA compression pruning. By leveraging the structural characteristics of Transformer modules, we design an approximate PCA method whose transformation matrices can be fully fused into the model without additional parameters. We also introduce a PCA-based global pruning ratio estimator that further considers the distribution of compressed activations, building on conventional module importance. We validate our method on Llama2, Llama3, and Phi series across various language benchmarks. Experimental results demonstrate that our approach achieves superior compression performance compared to recent baselines at the same compression ratio or inference speed.

Figures (10)

• Figure 1: Comparison of rank changes in layer outputs under PCA compression within modules (Block-intra) vs. between modules (Block-inter) on LLaMA2-7B, applying 50% sparsity at layers 5, 10, 15, 20, and 25. Lines indicate output ranks, reflecting activation distributions. Block-inter PCA continuously affects the distribution of subsequent activations, keeping them in a low-rank state. In contrast, Block-intra PCA has minimal impact on activation distributions and achieves better results. More detail can be found in Appendix \ref{['sec:appendixA']}.
• Figure 2: (a) The existing Inter-PCA pruning frameworks apply PCA compression between modules, which introduces additional computational overhead and error accumulation in residual paths. (b) Our IntraSlice framework (Intra-PCA) allows full fusion of matrices with less performance degradation. (c), (d) and (f) are the three components of IntraSlice, respectively.
• Figure 3: Schematic diagram of adaptive head PCA-based compression structure pruning and weight fusion.
• Figure 4: Inference speedup and memory of SoBP and IntraSlice at different sparsities, on the llama2-13B model.
• Figure 5: The impact of the size of calibration data and pruning bias $\lambda_b$ on PPL and zero-shot accuracy on Llama2-7B and Llama2-13B with 30% sparsity.