SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models

TL;DR

This work targets the memory bottleneck of KV caches in long-context LLMs by introducing SKVQ, a framework that combines channel reorder with clipped dynamic quantization and a sliding-window strategy to preserve recent KV caches in high precision. Offline calibration yields per-group quantization parameters, while attention-sink filters and windowing mitigate error accumulation, enabling ultra-low-bit quantization (2-bit keys, 1.5-bit values) with minimal accuracy loss across multiple models and tasks. SKVQ enables context lengths up to 1M on 80GB GPUs and promises substantial decoding-speedups, making long-context inference more practical at scale. This approach advances practical mixed-precision quantization for KV caches and provides a scalable path for deploying extremely long-context LLMs.

Abstract

Large language models (LLMs) can now handle longer sequences of tokens, enabling complex tasks like book understanding and generating lengthy novels. However, the key-value (KV) cache required for LLMs consumes substantial memory as context length increasing, becoming the bottleneck for deployment. In this paper, we present a strategy called SKVQ, which stands for sliding-window KV cache quantization, to address the issue of extremely low bitwidth KV cache quantization. To achieve this, SKVQ rearranges the channels of the KV cache in order to improve the similarity of channels in quantization groups, and applies clipped dynamic quantization at the group level. Additionally, SKVQ ensures that the most recent window tokens in the KV cache are preserved with high precision. This helps maintain the accuracy of a small but important portion of the KV cache.SKVQ achieves high compression ratios while maintaining accuracy. Our evaluation on LLMs demonstrates that SKVQ surpasses previous quantization approaches, allowing for quantization of the KV cache to 2-bit keys and 1.5-bit values with minimal loss of accuracy. With SKVQ, it is possible to process context lengths of up to 1M on an 80GB memory GPU for a 7b model and up to 7 times faster decoding.

Figures (7)

• Figure 1: Results on GovReport and MultiFieldQA-zh (Mistral-7b-Instruct-V0.2). We count the storage for meta data including quantization params and reorder index.
• Figure 2: Visualization of the key cache going through channel reorder and group clipping in sequence. The elements in the red/green box will be placed in the same group to share the quantization parameters.
• Figure 3: Overview of sliding window quantization Strategy. In each time step, we ensure the latest $w$ KV cache is full precision. For a token cache that slides out of the window, we make a decision based on the filter rules and choose whether to retain it to high precision.
• Figure 4: Average score on LongBench of SKVQ for Llama2-7b-chat and Mistral-7b-Instruct-V0.2.
• Figure 5: Comparison of SKVQ with KIVI on needle in haystack test. SKVQ achieved higher scores while using lower bitwidth.