Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SAES-SVD: Self-Adaptive Suppression of Accumulated and Local Errors for SVD-based LLM Compression

Xing Hu, Dawei Yang, Yuan Cheng, Zhixuan Chen, Zukang Xu

TL;DR

This paper tackles error accumulation in SVD-based LLM compression by introducing SAES-SVD, a framework that jointly optimizes intra-layer reconstruction and inter-layer error compensation. It combines Cumulative Error-Aware Layer Compression (CEALC), which adds a full-precision alignment term to the layer-wise objective, with Adaptive Collaborative Error Suppression (ACES), which adaptively tunes the compensation strength using second-order statistics. The approach yields a closed-form low-rank solution per layer and a principled, one-dimensional search for the adaptive parameter, enabling effective compression without any finetuning. Extensive experiments across models (including LLaMA variants) and tasks demonstrate consistent improvements over strong SVD-based baselines, including those requiring fine-tuning or mixed-rank schemes, and show practical gains in inference speed and memory efficiency. This work provides a scalable, training-free path for deploying large language models at low memory budgets while maintaining high fidelity to full-precision baselines.

Abstract

The rapid growth in the parameter scale of large language models (LLMs) has created a high demand for efficient compression techniques. As a hardware-agnostic and highly compatible technique, low-rank compression has been widely adopted. However, existing methods typically compress each layer independently by minimizing per-layer reconstruction error, overlooking a critical limitation: the reconstruction error propagates and accumulates through the network, which leads to amplified global deviations from the full-precision baseline. To address this, we propose Self-Adaptive Error Suppression SVD (SAES-SVD), a LLMs compression framework that jointly optimizes intra-layer reconstruction and inter-layer error compensation. SAES-SVD is composed of two novel components: (1) Cumulative Error-Aware Layer Compression (CEALC), which formulates the compression objective as a combination of local reconstruction and weighted cumulative error compensation. Based on it, we derive a closed-form low-rank solution relied on second-order activation statistics, which explicitly aligns each layer's output with its full-precision counterpart to compensate for accumulated errors. (2) Adaptive Collaborative Error Suppression (ACES), which automatically adjusts the weighting coefficient to enhance the low-rank structure of the compression objective in CEALC. Specifically, the coefficient is optimized to maximize the ratio between the Frobenius norm of the compressed layer's output and that of the compression objective under a fixed rank, thus ensuring that the rank budget is utilized effectively. Extensive experiments across multiple LLM architectures and tasks show that, without fine-tuning or mixed-rank strategies, SAES-SVD consistently improves post-compression performance.

SAES-SVD: Self-Adaptive Suppression of Accumulated and Local Errors for SVD-based LLM Compression

TL;DR

This paper tackles error accumulation in SVD-based LLM compression by introducing SAES-SVD, a framework that jointly optimizes intra-layer reconstruction and inter-layer error compensation. It combines Cumulative Error-Aware Layer Compression (CEALC), which adds a full-precision alignment term to the layer-wise objective, with Adaptive Collaborative Error Suppression (ACES), which adaptively tunes the compensation strength using second-order statistics. The approach yields a closed-form low-rank solution per layer and a principled, one-dimensional search for the adaptive parameter, enabling effective compression without any finetuning. Extensive experiments across models (including LLaMA variants) and tasks demonstrate consistent improvements over strong SVD-based baselines, including those requiring fine-tuning or mixed-rank schemes, and show practical gains in inference speed and memory efficiency. This work provides a scalable, training-free path for deploying large language models at low memory budgets while maintaining high fidelity to full-precision baselines.

Abstract

The rapid growth in the parameter scale of large language models (LLMs) has created a high demand for efficient compression techniques. As a hardware-agnostic and highly compatible technique, low-rank compression has been widely adopted. However, existing methods typically compress each layer independently by minimizing per-layer reconstruction error, overlooking a critical limitation: the reconstruction error propagates and accumulates through the network, which leads to amplified global deviations from the full-precision baseline. To address this, we propose Self-Adaptive Error Suppression SVD (SAES-SVD), a LLMs compression framework that jointly optimizes intra-layer reconstruction and inter-layer error compensation. SAES-SVD is composed of two novel components: (1) Cumulative Error-Aware Layer Compression (CEALC), which formulates the compression objective as a combination of local reconstruction and weighted cumulative error compensation. Based on it, we derive a closed-form low-rank solution relied on second-order activation statistics, which explicitly aligns each layer's output with its full-precision counterpart to compensate for accumulated errors. (2) Adaptive Collaborative Error Suppression (ACES), which automatically adjusts the weighting coefficient to enhance the low-rank structure of the compression objective in CEALC. Specifically, the coefficient is optimized to maximize the ratio between the Frobenius norm of the compressed layer's output and that of the compression objective under a fixed rank, thus ensuring that the rank budget is utilized effectively. Extensive experiments across multiple LLM architectures and tasks show that, without fine-tuning or mixed-rank strategies, SAES-SVD consistently improves post-compression performance.
Paper Structure (60 sections, 2 theorems, 55 equations, 6 figures, 11 tables, 3 algorithms)

This paper contains 60 sections, 2 theorems, 55 equations, 6 figures, 11 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Large Language Model Compression Techniques.
  4. SVD-based Compression for LLMs.
  5. Preliminaries
  6. Method
  7. Cumulative Error- Aware Layer Compression
  8. Layer compression objective with cumulative error awareness
  9. Eliminating explicit inputs via second-order statistics.
  10. Closed-form via truncated SVD.
  11. Adaptive Collaborative Error Suppression
  12. Experiments.
  13. Models and Datasets.
  14. Baselines.
  15. Main Results.
  16. ...and 45 more sections

Key Result

Theorem 4.1

As illustrated by the detailed derivation in the Appendix theorem1 derivation, let $H_{\ell}=X_\ell X_\ell^\top\succ0$ and $L_\ell=H_{\ell}^{-1/2}$. Then, for any rank constraint $r_\ell$,

Figures (6)

  • Figure 1: (a) Cumulative error analysis on LLaMA-3-8B, showing the cosine similarity of compressed outputs with the original reference across layers. SAES-SVD effectively mitigates error accumulation by adaptively suppressing upstream deviations. (b) Perplexity on WikiText2 under varying compression ratios. SAES-SVD consistently achieves lower perplexity than ASVD and SVD-LLM. (c) Retained Energy Ratio achieved by using different weighting coefficients for the cumulative error compensation term in with rank=128 compression for k-proj in the 14th Layer of LLaMA-3-8B. Here, for the sake of convenience, we use $\beta=\alpha/(1+\alpha)$ as the horizontal axis. (d) Effect of fixed versus adaptive weighting coefficients for the cumulative error compensation on average accuracy across 7 zero-shot benchmarks.
  • Figure 2: Overview of the proposed SAES-SVD framework.
  • Figure 3: Wiki-PPL for two large-scale models, across different methods at 20% compression.
  • Figure 4: Memory usage and inference speedup of LLaMA-3-8B on varying compression ratios.
  • Figure 5: Using different beta coefficients for the k-proj of LLama3-8B at layer 14, layer 3 and layer 20 after the SVD in Equation \ref{['eq:final svd']}, the proportion of retained energy at rank-1228.
  • ...and 1 more figures

Theorems & Definitions (2)

  • Theorem 4.1
  • Theorem 4.2: First-Order Approximation of Fixed Subspace(FS-FOA)