Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression

TL;DR

Zero Sum SVD introduces a post-training SVD-based compression for LLMs that selects singular components across layers under a global loss-budget using activation whitening and first-order calibration loss estimates. A zero-sum greedy rule preserves the cumulative predicted loss drift near zero, automatically yielding heterogeneous per-layer ranks without expensive rank-allocation optimization; an optional projected-gradient correction further boosts performance while keeping the rank low. The method demonstrates consistent perplexity and accuracy gains across diverse model sizes and benchmarks, with additional practical benefits in inference throughput and memory efficiency. Its hardware-agnostic, post-training nature makes it attractive for deploying capable LLMs on resource-constrained devices while maintaining accuracy and speed.

Abstract

Advances in large language models have driven strong performance across many tasks, but their memory and compute costs still hinder deployment. SVD-based compression reduces storage and can speed up inference via low-rank factors, yet performance depends on how rank is allocated under a global compression ratio. Prior methods often use homogeneous ranks for similarly sized matrices, despite large differences in loss sensitivity, or rely on expensive iterative pre-truncation optimization to determine per matrix ranks. We propose \textbf{Zero Sum SVD} (\textbf{ZS-SVD}), a post-training method that performs \emph{global} singular component selection using activation whitening and first-order calibration loss estimates in whitened coordinates. \textbf{ZS-SVD} prunes components across the whole model with a \textbf{zero sum} rule that keeps the cumulative predicted loss change near zero, automatically yielding heterogeneous ranks without solving a rank allocation optimization. Motivated by evidence that gradients near pretrained solutions exhibit low rank structure, we also introduce an optional lightweight correction that applies a \textbf{single} projected gradient update after truncation, followed by re-truncation. Extensive experiments across multiple LLM architectures show consistent gains across diverse benchmarks and compression ratios. Code is available at https://github.com/mint-vu/Zero-Sum-SVD

Figures (4)

• Figure 1: Zero-Sum SVD scores singular values of the whitened weight matrix using the calibration-loss directional derivative, then uses these scores to prioritize truncation across layers.
• Figure 2: Zero-sum selection rule across different weight matrices.
• Figure 3: The numbers above each bar indicate the effective-rank ratio $k_{0.95}(G)/k_{0.95}(W')$ (grad over weight).
• Figure 4: Gradient vs. weight effective rank at Layers 1, 16, and 32 of LLaMA-2-7B under 20% pruning. For each module, we compute the singular spectrum of the truncated weight $W'$ and the gradient $G=\nabla_W\mathcal{L}(W')$, and define the effective rank using a spectral energy cutoff of 0.95 (vertical dashed line), i.e., the smallest $k$ such that $\sum_{i\le k}\sigma_i^2/\sum_i\sigma_i^2 \ge 0.95$.

Theorems & Definitions (5)

• Theorem 3.1: Whitened truncation yields singular value reconstruction loss
• Corollary 3.2: Optimality of truncated SVD in the whitened space
• Lemma 4.1: Rank bound under additive correction
• proof : Proof of Theorem \ref{['thm:whitened_loss']}
• proof : Proof of Corollary \ref{['cor:optimal_trunc']}