Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMs

Jian Chen, Zhuoran Wang, Jiayu Qin, Ming Li, Meng Wang, Changyou Chen, Yin Chen, Qizhen Weng, Yirui Liu

TL;DR

KV-CoRE introduces a data-dependent framework for evaluating KV-cache compressibility in LLMs using an incremental SVD to compute globally optimal low-rank approximations under the Frobenius norm. The approach yields per-layer, dataset-level insights via the Normalized Effective Rank, which correlates with end-to-end performance metrics such as perplexity and GPT-based quality assessments. A large-scale benchmark reveals consistent patterns where keys are more compressible than values, cross-lingual variation dominates cross-domain shifts, and KV capacity shapes compressibility, with rank collapse observed in under-represented languages. The work establishes a principled evaluation framework and demonstrates the potential for dynamic, data-aware KV-cache compression and data-centric model improvements across multilingual and domain-rich settings.

Abstract

Large language models rely on kv-caches to avoid redundant computation during autoregressive decoding, but as context length grows, reading and writing the cache can quickly saturate GPU memory bandwidth. Recent work has explored KV-cache compression, yet most approaches neglect the data-dependent nature of kv-caches and their variation across layers. We introduce KV-CoRE KV-cache Compressibility by Rank Evaluation), an SVD-based method for quantifying the data-dependent low-rank compressibility of kv-caches. KV-CoRE computes the optimal low-rank approximation under the Frobenius norm and, being gradient-free and incremental, enables efficient dataset-level, layer-wise evaluation. Using this method, we analyze multiple models and datasets spanning five English domains and sixteen languages, uncovering systematic patterns that link compressibility to model architecture, training data, and language coverage. As part of this analysis, we employ the Normalized Effective Rank as a metric of compressibility and show that it correlates strongly with performance degradation under compression. Our study establishes a principled evaluation framework and the first large-scale benchmark of kv-cache compressibility in LLMs, offering insights for dynamic, data-aware compression and data-centric model development.

KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMs

TL;DR

KV-CoRE introduces a data-dependent framework for evaluating KV-cache compressibility in LLMs using an incremental SVD to compute globally optimal low-rank approximations under the Frobenius norm. The approach yields per-layer, dataset-level insights via the Normalized Effective Rank, which correlates with end-to-end performance metrics such as perplexity and GPT-based quality assessments. A large-scale benchmark reveals consistent patterns where keys are more compressible than values, cross-lingual variation dominates cross-domain shifts, and KV capacity shapes compressibility, with rank collapse observed in under-represented languages. The work establishes a principled evaluation framework and demonstrates the potential for dynamic, data-aware KV-cache compression and data-centric model improvements across multilingual and domain-rich settings.

Abstract

Large language models rely on kv-caches to avoid redundant computation during autoregressive decoding, but as context length grows, reading and writing the cache can quickly saturate GPU memory bandwidth. Recent work has explored KV-cache compression, yet most approaches neglect the data-dependent nature of kv-caches and their variation across layers. We introduce KV-CoRE KV-cache Compressibility by Rank Evaluation), an SVD-based method for quantifying the data-dependent low-rank compressibility of kv-caches. KV-CoRE computes the optimal low-rank approximation under the Frobenius norm and, being gradient-free and incremental, enables efficient dataset-level, layer-wise evaluation. Using this method, we analyze multiple models and datasets spanning five English domains and sixteen languages, uncovering systematic patterns that link compressibility to model architecture, training data, and language coverage. As part of this analysis, we employ the Normalized Effective Rank as a metric of compressibility and show that it correlates strongly with performance degradation under compression. Our study establishes a principled evaluation framework and the first large-scale benchmark of kv-cache compressibility in LLMs, offering insights for dynamic, data-aware compression and data-centric model development.
Paper Structure (31 sections, 8 equations, 9 figures, 1 table)

This paper contains 31 sections, 8 equations, 9 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Rank Analysis in Language Models:
  4. Low-Rank KV-cache Compression:
  5. Experiment
  6. Models
  7. Datasets
  8. Hardware and Software Setup
  9. Evaluation Metric
  10. Normalized Effective Rank
  11. Perplexity
  12. Normalized Delta-Perplexity
  13. GPT Score
  14. Benchmarking Experiment on KV-Cache Compressibility
  15. Average NER across Models and Datasets
  16. ...and 16 more sections

Figures (9)

  • Figure 1: Layer-wise NER of key and value representations in Qwen3-4B, evaluated on 5 datasets and 3 languages from the VisR-Bench benchmark.
  • Figure 2: PPL heatmap of Qwen3-4B and LLaMA-2-7b on the Alpaca dataset.
  • Figure 3: GPT score heatmap of Qwen3-4B and LLaMA-2-7b on the Alpaca dataset.
  • Figure 4: Correlation between dataset-level NER and ND-PPL computed by Qwen3-4B.
  • Figure B.1: Layer-wise NER of key and value representations in Qwen3-4B evaluated on 5 datasets and 3 languages from the VisR-Bench benchmark.
  • ...and 4 more figures