Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption

Luohe Shi, Hongyi Zhang, Yao Yao, Zuchao Li, Hai Zhao

TL;DR

This review surveys KV-Cache optimization for long-context LLMs, spanning training-time architectural changes, deployment-time system frameworks, and post-training eviction, merging, and quantization strategies. It highlights methods to shrink KV size via multi-query and grouped-query attention, cross-layer reuse, and context compression, while examining deployment frameworks (Paged/Dist Attention) and cross-dialog reuse to improve throughput. Evaluation relies on long-context benchmarks, key-retrieval tasks, and few-shot tests, using metrics for memory, throughput, latency, and perplexity to balance efficiency with capability. The paper argues for a holistic approach that combines memory-centric techniques with robust evaluation to guide practical, scalable LLM deployment.

Abstract

Large Language Models (LLMs), epitomized by ChatGPT's release in late 2022, have revolutionized various industries with their advanced language comprehension. However, their efficiency is challenged by the Transformer architecture's struggle with handling long texts. KV Cache has emerged as a pivotal solution to this issue, converting the time complexity of token generation from quadratic to linear, albeit with increased GPU memory overhead proportional to conversation length. With the development of the LLM community and academia, various KV Cache compression methods have been proposed. In this review, we dissect the various properties of KV Cache and elaborate on various methods currently used to optimize the KV Cache space usage of LLMs. These methods span the pre-training phase, deployment phase, and inference phase, and we summarize the commonalities and differences among these methods. Additionally, we list some metrics for evaluating the long-text capabilities of large language models, from both efficiency and capability perspectives. Our review thus sheds light on the evolving landscape of LLM optimization, offering insights into future advancements in this dynamic field. Links to the papers mentioned in this review can be found in our Github Repo https://github.com/zcli-charlie/Awesome-KV-Cache.

Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption

TL;DR

This review surveys KV-Cache optimization for long-context LLMs, spanning training-time architectural changes, deployment-time system frameworks, and post-training eviction, merging, and quantization strategies. It highlights methods to shrink KV size via multi-query and grouped-query attention, cross-layer reuse, and context compression, while examining deployment frameworks (Paged/Dist Attention) and cross-dialog reuse to improve throughput. Evaluation relies on long-context benchmarks, key-retrieval tasks, and few-shot tests, using metrics for memory, throughput, latency, and perplexity to balance efficiency with capability. The paper argues for a holistic approach that combines memory-centric techniques with robust evaluation to guide practical, scalable LLM deployment.

Abstract

Large Language Models (LLMs), epitomized by ChatGPT's release in late 2022, have revolutionized various industries with their advanced language comprehension. However, their efficiency is challenged by the Transformer architecture's struggle with handling long texts. KV Cache has emerged as a pivotal solution to this issue, converting the time complexity of token generation from quadratic to linear, albeit with increased GPU memory overhead proportional to conversation length. With the development of the LLM community and academia, various KV Cache compression methods have been proposed. In this review, we dissect the various properties of KV Cache and elaborate on various methods currently used to optimize the KV Cache space usage of LLMs. These methods span the pre-training phase, deployment phase, and inference phase, and we summarize the commonalities and differences among these methods. Additionally, we list some metrics for evaluating the long-text capabilities of large language models, from both efficiency and capability perspectives. Our review thus sheds light on the evolving landscape of LLM optimization, offering insights into future advancements in this dynamic field. Links to the papers mentioned in this review can be found in our Github Repo https://github.com/zcli-charlie/Awesome-KV-Cache.
Paper Structure (25 sections, 3 equations, 5 figures, 2 tables)

This paper contains 25 sections, 3 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Preliminary and Notations
  3. Training Stage Optimization
  4. Deploy-Stage Optimization
  5. Post-Training Optimizations
  6. Eviction and Merging
  7. Quantization
  8. KV Cache only quantization
  9. Full quantization
  10. Evaluation
  11. Datasets
  12. Long context benches
  13. Key retrieval
  14. Few-shot Testing
  15. Evaluation Metric
  16. ...and 10 more sections

Figures (5)

  • Figure 1: An overview of the main structure of KV Cache compression methods
  • Figure 2: The comparasion between MHA, MQA and GQA. To be note that the final linear layer $W_{o}^{(l)}$ is not depicted here.
  • Figure 3: The comparison between different KV Cache policies
  • Figure 4: Prompt format for passkey retrieval. ($\textless$PASS KEY$\textgreater$ is a 5-digit number.)
  • Figure 5: Example for Needle in a Haystack task. Facts are inserted in the sentences of irrelevant text