Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelity

Damien Lesens, Beheshteh T. Rakhshan, Guillaume Rabusseau

TL;DR

KQ-SVD introduces a provably optimal, interaction-aware low-rank compression of the KV cache for transformer attention. By directly optimizing the QK interaction with a closed-form factorization A = K^+ Û, B = K^T Û, it preserves attention fidelity better than key-only or concatenated QK methods. The approach extends to Grouped-Query Attention and is supported by theoretical gaps and empirical gains across multiple LLMs on the C4 dataset. This enables memory-efficient inference without retraining, improving practicality for long-context generation. Overall, KQ-SVD advances KV-cache compression by aligning the low-rank approximation with the fundamental inner-product structure of attention.

Abstract

The Key-Value (KV) cache is central to the efficiency of transformer-based large language models (LLMs), storing previously computed vectors to accelerate inference. Yet, as sequence length and batch size grow, the cache becomes a major memory bottleneck. Prior compression methods typically apply low-rank decomposition to keys alone or attempt to jointly embed queries and keys, but both approaches neglect that attention fundamentally depends on their inner products. In this work, we prove that such strategies are suboptimal for approximating the attention matrix. We introduce KQ-SVD, a simple and computationally efficient method that directly performs an optimal low-rank decomposition of the attention matrix via a closed-form solution. By targeting the true source of redundancy, KQ-SVD preserves attention outputs with higher fidelity under compression. Extensive evaluations on LLaMA and Mistral models demonstrate that our approach consistently delivers superior projection quality.

KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelity

TL;DR

KQ-SVD introduces a provably optimal, interaction-aware low-rank compression of the KV cache for transformer attention. By directly optimizing the QK interaction with a closed-form factorization A = K^+ Û, B = K^T Û, it preserves attention fidelity better than key-only or concatenated QK methods. The approach extends to Grouped-Query Attention and is supported by theoretical gaps and empirical gains across multiple LLMs on the C4 dataset. This enables memory-efficient inference without retraining, improving practicality for long-context generation. Overall, KQ-SVD advances KV-cache compression by aligning the low-rank approximation with the fundamental inner-product structure of attention.

Abstract

The Key-Value (KV) cache is central to the efficiency of transformer-based large language models (LLMs), storing previously computed vectors to accelerate inference. Yet, as sequence length and batch size grow, the cache becomes a major memory bottleneck. Prior compression methods typically apply low-rank decomposition to keys alone or attempt to jointly embed queries and keys, but both approaches neglect that attention fundamentally depends on their inner products. In this work, we prove that such strategies are suboptimal for approximating the attention matrix. We introduce KQ-SVD, a simple and computationally efficient method that directly performs an optimal low-rank decomposition of the attention matrix via a closed-form solution. By targeting the true source of redundancy, KQ-SVD preserves attention outputs with higher fidelity under compression. Extensive evaluations on LLaMA and Mistral models demonstrate that our approach consistently delivers superior projection quality.

Paper Structure

This paper contains 25 sections, 6 theorems, 44 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Related Works
  3. Low-rank structure of the KV-cache.
  4. KV Weights Compression.
  5. Positioning of KQ-SVD.
  6. Preliminary
  7. Notations
  8. Background
  9. Cache compression with SVD
  10. Cache compression with Eigen
  11. Methodology
  12. Motivation
  13. Proposed Method
  14. KQ-SVD: Optimal attention factorization
  15. Theoretical analysis
  16. ...and 10 more sections

Key Result

Theorem 1

Let $\mathbf{X}\in\mathbb{R}^{T\times d}$ be a sequence of token embeddings, $\mathbf{K},\mathbf{Q}, \mathbf{V}\in\mathbb{R}^{T\times d}$ and where $\widetilde{\mathrm{MHA}}(\mathbf{X}), \widetilde{\mathbf{K}_i}$ and $\widetilde{\mathbf{V}_i}$ represent the approximation of $\mathrm{MHA}(\mathbf{X}),\mathbf{K}_i$ and $\mathbf{V}_i$, respectively. The difference between the actual attention output

Figures (2)

  • Figure 1: Relative Frobenius approximation output error per layer (top) and mean relative errors on $\mathbf{K}$, $\mathbf{Q}$, $\mathbf{V}$, $\mathbf{K}\mathbf{Q}^\top$ and attention layer output across layers (bottom) for Mistral and Llama models.
  • Figure 2: Llama2-7B relative output approximation error averaged across layers for varying unbalanced factor $\beta$.

Theorems & Definitions (6)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Theorem 5
  • Theorem