CPSVD: Enhancing Large Language Model Compression via Column-Preserving Singular Value Decomposition

TL;DR

The paper tackles inefficiencies in uniform SVD-based compression for large language models by recognizing non-uniform truncation errors across matrix columns. It proposes CPSVD, which preserves high-error columns and applies SVD to the rest, and further allocates layer-wide compression adaptively at the module level based on measured reconstruction errors. Empirical results across multiple models, datasets, and compression rates show CPSVD consistently outperforms state-of-the-art SVD-based methods in perplexity and zero-shot accuracy, with favorable efficiency characteristics. This approach provides a scalable, architecture-agnostic strategy for more effective SVD-based LLM compression and has potential applicability to other column-dependent decomposition methods.

Abstract

The rapid advancement of Large Language Models (LLMs) faces a critical bottleneck in their immense size, necessitating efficient compression techniques. While Singular Value Decomposition (SVD) is a promising approach, existing SVD-based methods treat the entire parameter matrix uniformly, overlooking that SVD approximation errors vary significantly across different matrix parts, which often leads to suboptimal compression. To address this, we propose \textbf{C}olumn-\textbf{P}reserving \textbf{S}ingular \textbf{V}alue \textbf{D}ecomposition (CPSVD), a novel method that refines SVD-based LLM compression by intelligently segmenting the parameter matrix. Unlike traditional SVD, CPSVD identifies and directly preserves matrix columns with high decomposition errors, applying SVD only to columns with low decomposition errors, while precisely determining the optimal balance point between these two strategies to minimize error. Furthermore, leveraging the inherent heterogeneity in decomposition errors across different matrices within an LLM, CPSVD adaptively allocates non-uniform compression rates to modules within that layer, while adhering to a target layer-wise compression ratio, thereby further enhancing compression performance. Extensive experiments demonstrate that CPSVD consistently outperforms state-of-the-art SVD-based LLM compression methods, achieving lower perplexity and higher accuracy on zero-shot tasks.

Figures (5)

• Figure 1: Workflow of CPSVD. (a) illustrates the module compression rate allocation: it first computes individual losses at a uniform compression rate, then determines each module's rate based on relative loss and the layer's overall compression ratio. (b) presents the column-preserving strategy: a workflow that includes decomposition, calculating and ranking column errors, and searching for the optimal number of columns to retain.
• Figure 2: Column Truncation Error for Different Modules
• Figure 3: Normalized Loss vs. Preserved Column Number for Various Modules in LLaMA 7B Model's Layer 0 Under 20% Compression Ratio
• Figure 4: Relative Loss vs. Layer Index for Different Modules in LLaMA 7B Model at 20% Compression Ratio
• Figure 5: Throughput & Memory Usage of Compressed LLaMA-7B (Batch Size=4, Seq Length=1024).