Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization

Yixuan Wang, Qingyu Shi, Jiayu Zhou, Dianbo Liu, Ziwei He, Zhouhan Lin

Abstract

The growing context length of Large Language Models (LLMs) enlarges the Key-Value (KV) cache, limiting deployment in resource-limited environments. Prior training-free approaches for KV cache compression typically rely on low-rank approximation or scalar quantization, which fail to simultaneously achieve high compression ratios and high reconstruction fidelity. We propose VQKV, a novel, training-free method introducing vector quantization (VQ) to obtain highly compressed KV representations while preserving high model fidelity, allowing for the representation of thousands of floating-point values with just a few integer indices. As a result, VQKV achieves an 82.8\% compression ratio on LLaMA3.1-8B while retaining 98.6\% of the baseline performance on LongBench and enabling 4.3x longer generation length on the same memory footprint.

VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization

Abstract

The growing context length of Large Language Models (LLMs) enlarges the Key-Value (KV) cache, limiting deployment in resource-limited environments. Prior training-free approaches for KV cache compression typically rely on low-rank approximation or scalar quantization, which fail to simultaneously achieve high compression ratios and high reconstruction fidelity. We propose VQKV, a novel, training-free method introducing vector quantization (VQ) to obtain highly compressed KV representations while preserving high model fidelity, allowing for the representation of thousands of floating-point values with just a few integer indices. As a result, VQKV achieves an 82.8\% compression ratio on LLaMA3.1-8B while retaining 98.6\% of the baseline performance on LongBench and enabling 4.3x longer generation length on the same memory footprint.
Paper Structure (17 sections, 13 equations, 8 figures, 4 tables)

This paper contains 17 sections, 13 equations, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. KV Cache Compression
  4. Vector Quantization
  5. Compressing KV with Vector Quantization
  6. Learning High-fidelity VQ Codebooks
  7. Prefilling Stage
  8. Decoding Stage
  9. Compression Ratio
  10. Experiment
  11. Setup
  12. Long-Context Evaluation
  13. Memory Efficiency
  14. Ablation Study
  15. Conclusion
  16. ...and 2 more sections

Figures (8)

  • Figure 1: Overview of our VQKV. The left part shows the detailed process of our VQKV on prefilling stage and decoding stage. The right part shows the overview of our VQKV.
  • Figure 2: Results of LLaMA3.1-8B on NIAH.
  • Figure 3: Results of LLaMA3.2-3B on NIAH.
  • Figure 4: Detailed results of LLaMA3.1-8B on RULER in different context length. The results of full cache model are taken as the 100% reference.
  • Figure 5: Detailed results of LLaMA3.2-3B on RULER in different context length. The results of full cache model are taken as the 100% reference.
  • ...and 3 more figures