Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

QAQ: Quality Adaptive Quantization for LLM KV Cache

Shichen Dong, Wen Cheng, Jiayu Qin, Wei Wang

TL;DR

<3-5 sentence high-level summary> QAQ addresses the KV cache memory bottleneck in long-context LLMs by introducing a quality adaptive quantization that treats key and value caches separately, accounts for outliers with mixed-precision, and employs an attention-window to handle exceptional tokens. It provides a theoretical framework for per-token bit allocation under variance bounds and derives practical quantization strategies, including attention-score prediction. Empirical results on LLaMA 2 7B/13B show up to ~10x KV-cache compression with negligible accuracy drop and outperform existing KV-cache quantization methods. This work substantially enhances the practicality of deploying long-context LLMs by dramatically reducing memory footprints while preserving performance, and it offers publicly available code for replication.

Abstract

The emergence of LLMs has ignited a fresh surge of breakthroughs in NLP applications, particularly in domains such as question-answering systems and text generation. As the need for longer context grows, a significant bottleneck in model deployment emerges due to the linear expansion of the Key-Value (KV) cache with the context length. Existing methods primarily rely on various hypotheses, such as sorting the KV cache based on attention scores for replacement or eviction, to compress the KV cache and improve model throughput. However, heuristics used by these strategies may wrongly evict essential KV cache, which can significantly degrade model performance. In this paper, we propose QAQ, a Quality Adaptive Quantization scheme for the KV cache. We theoretically demonstrate that key cache and value cache exhibit distinct sensitivities to quantization, leading to the formulation of separate quantization strategies for their non-uniform quantization. Through the integration of dedicated outlier handling, as well as an improved attention-aware approach, QAQ achieves up to 10x the compression ratio of the KV cache size with a neglectable impact on model performance. QAQ significantly reduces the practical hurdles of deploying LLMs, opening up new possibilities for longer-context applications. The code is available at github.com/ClubieDong/KVCacheQuantization.

QAQ: Quality Adaptive Quantization for LLM KV Cache

TL;DR

<3-5 sentence high-level summary> QAQ addresses the KV cache memory bottleneck in long-context LLMs by introducing a quality adaptive quantization that treats key and value caches separately, accounts for outliers with mixed-precision, and employs an attention-window to handle exceptional tokens. It provides a theoretical framework for per-token bit allocation under variance bounds and derives practical quantization strategies, including attention-score prediction. Empirical results on LLaMA 2 7B/13B show up to ~10x KV-cache compression with negligible accuracy drop and outperform existing KV-cache quantization methods. This work substantially enhances the practicality of deploying long-context LLMs by dramatically reducing memory footprints while preserving performance, and it offers publicly available code for replication.

Abstract

The emergence of LLMs has ignited a fresh surge of breakthroughs in NLP applications, particularly in domains such as question-answering systems and text generation. As the need for longer context grows, a significant bottleneck in model deployment emerges due to the linear expansion of the Key-Value (KV) cache with the context length. Existing methods primarily rely on various hypotheses, such as sorting the KV cache based on attention scores for replacement or eviction, to compress the KV cache and improve model throughput. However, heuristics used by these strategies may wrongly evict essential KV cache, which can significantly degrade model performance. In this paper, we propose QAQ, a Quality Adaptive Quantization scheme for the KV cache. We theoretically demonstrate that key cache and value cache exhibit distinct sensitivities to quantization, leading to the formulation of separate quantization strategies for their non-uniform quantization. Through the integration of dedicated outlier handling, as well as an improved attention-aware approach, QAQ achieves up to 10x the compression ratio of the KV cache size with a neglectable impact on model performance. QAQ significantly reduces the practical hurdles of deploying LLMs, opening up new possibilities for longer-context applications. The code is available at github.com/ClubieDong/KVCacheQuantization.
Paper Structure (18 sections, 13 equations, 4 figures, 3 tables)

This paper contains 18 sections, 13 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Problem Description and Related Work
  3. Problem Description
  4. Related Work
  5. Related Work
  6. Insights
  7. Key Cache and Value Cache Exhibit Distinct Sensitivities to Quantization
  8. Persistence of Importance has Exceptions
  9. Outliers in KV Cache Matter
  10. Quality Adaptive Quantization Method for Key Cache and Value Cache
  11. Quantitative Formula Derivation
  12. Methods
  13. Evaluation
  14. Experiment Settings
  15. Compression Ratio vs. Accuracy
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Calculation process of quantized key and value vectors, $\frac{1}{\sqrt{D}}$ is neglected.
  • Figure 2: The experiment demonstration of our key insights.
  • Figure 3: Experiment performance of QAQ.
  • Figure 4: Experiment result of QAQ v.s. non-attention quantization.