Large Language Model Compression with Global Rank and Sparsity Optimization

TL;DR

This work tackles robust compression of pre-trained Transformer-based LLMs by proposing CAP, a two-stage framework that first uses RPCA to decompose weight matrices into a global low-rank component $L$ and a sparse residual $S$ (i.e., $W = L + S$), and then applies a learnable, policy-gradient-based Bernoulli pruning to jointly select retained directions in $L$ and nonzero entries in $S$ under a parameter budget. The method automatically adapts the rank and sparsity across layers, avoiding manual thresholds and extensive fine-tuning. Empirical results on LLaMA and LLaMA-2 models show CAP outperforms state-of-the-art sparsification and composite-approximation baselines at 50% compression, with favorable efficiency and generality across architectures. The approach offers a practical, hardware-aware pathway to deploy large models under constrained resources, and it opens avenues for integration with quantization or distillation for even larger-scale deployments.

Abstract

Low-rank and sparse composite approximation is a natural idea to compress Large Language Models (LLMs). However, such an idea faces two primary challenges that adversely affect the performance of existing methods. The first challenge relates to the interaction and cooperation between low-rank and sparse matrices, while the second involves determining weight allocation across different layers, as redundancy varies considerably among them. To address these challenges, we propose a novel two-stage LLM compression method with the capability of global rank and sparsity optimization. It is noteworthy that the overall optimization space is vast, making comprehensive optimization computationally prohibitive. Therefore, to reduce the optimization space, our first stage utilizes robust principal component analysis to decompose the weight matrices of LLMs into low-rank and sparse components, which span the low dimensional and sparse spaces containing the resultant low-rank and sparse matrices, respectively. In the second stage, we propose a probabilistic global optimization technique to jointly identify the low-rank and sparse structures within the above two spaces. The appealing feature of our approach is its ability to automatically detect the redundancy across different layers and to manage the interaction between the sparse and low-rank components. Extensive experimental results indicate that our method significantly surpasses state-of-the-art techniques for sparsification and composite approximation.

Figures (5)

• Figure 1: Singular values of Layer 0 and Layer 31 across different modules, comparing original and RPCA-processed matrices. The dotted line represents the singular value distribution of the original model, and the solid line represents the singular value distribution of the low-rank matrix after RPCA processing.
• Figure 2: Overview of our proposed compression method. The weight matrix $\mathbf{W}$ is decomposed into a low-rank component $\mathbf{L}$ and a sparse component $\mathbf{S}$ using RPCA. Both components are pruned through Bernoulli sampling guided by learned probability scores, optimized via policy gradient. The low-rank component is further factorized into $\mathbf{U}'$ and $\mathbf{V}'$ to reduce the number of model para meters.
• Figure 3: Perplexity changes with sequential decomposition and pruning of each layer in LLaMA2-7B.
• Figure 4: The rank distribution of low-rank matrices for each module after pruning.
• Figure 5: An illustration of how factual knowledge is encoded and attributed within Transformer architectures. Factual knowledge is distributed across feed-forward networks (FFNs) and attention mechanisms. Pruning these components risks disrupting knowledge structures, leading to performance degradation.

Theorems & Definitions (4)

• Theorem F.1: Low-Rank+Sparse Approximation
• proof
• Lemma F.2: Parameter Efficiency
• Corollary F.3: LLM Performance Preservation