SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attention
Hong Yankun, Li Xing, Zhen Hui-Ling, Yu Xianzhi, Liu Wulong, Yuan Mingxuan
TL;DR
This work tackles the KV-cache memory bottleneck in LLM inference by introducing SVDq, a mixed-precision quantization scheme that projects KV caches into latent channels via Singular Value Decomposition and allocates higher precision to channels associated with larger singular values. The method combines SVD-based channel compression with per-channel quantization to achieve up to $410\times$ compression at a $1.25$-bit equivalent, while maintaining or exceeding the performance of direct quantization on benchmarks like RULER and LongBench. Theoretical analysis shows quantization error under SVDq is significantly lower than traditional per-channel quantization due to the exponential decay of latent-channel variances, and empirical results demonstrate strong compatibility with sparsity (ShadowKV) and optional $V$-cache quantization. Overall, SVDq offers a scalable, high-precision, low-memory KV-cache compression strategy that can dramatically reduce memory footprint during inference with minimal degradation in accuracy.
Abstract
For the efficient inference of Large Language Models (LLMs), the effective compression of key-value (KV) cache is essential. Three main types of KV cache compression techniques, namely sparsity, channel compression, and quantization, have been identified. This study presents SVDq, a Singular Value Decomposition (SVD) - based mixed precision quantization method for K cache. Initially, K cache is transformed into latent channels using SVD basis representations. Since the values in latent channels decay rapidly and become negligible after only a few latent channels, our method then incorporates importance-aware quantization and compression for latent channels. This enables the effective allocation of higher precision to more significant channels. Theoretically, we prove that SVDq results in quantization errors (x0.1 or even lower) that are much lower than those of per-channel key quantization in the original space. Our findings based on RULER and LongBench benchmarks demonstrate that SVDq can achieve an equivalent key cache precision as low as 1.25-bit. When combined with key sparsity, it can reach a key compression ratio of up to 410x for attention computation, all while maintaining comparable model performance. Notably, our method is nearly lossless for LongBench datasets. This indicates that SVDq enables high-precision low-bit quantization, providing a more efficient solution for KV cache compression in LLMs.