Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

More Than a Quick Glance: Overcoming the Greedy Bias in KV-Cache Compression

Aryan Sood, Tanvi Sharma, Vansh Agrawal

TL;DR

The paper tackles the challenge of long-context memory under fixed KV-cache budgets in LLMs. It introduces LASER-KV, a block-wise, accumulative KV memory with a protection divisor $n$ and an Exact-LSH recall policy that fuses $ExactAttention$ and LSH for token selection. On Babilong, standard compression degrades performance by about $15\%$ to $30\%$ at long contexts, while LASER-KV remains stable up to $128k$ tokens and achieves up to $10\%$ accuracy gains at $128k$. The results argue that token utility in long contexts cannot be captured by attention scores alone and highlight the value of hybrid, structure-aware selection for sustainable long-term memory in LLMs.

Abstract

While Large Language Models (LLMs) can theoretically support extensive context windows, their actual deployment is constrained by the linear growth of Key-Value (KV) cache memory. Prevailing compression strategies mitigate this through various pruning mechanisms, yet trade-off semantic recall for memory efficiency. In this work, we present LASER-KV (Layer Accumulated Selection with Exact-LSH Recall), a framework designed to test the limits of KV compression under a strict accumulative budgeting policy. We deviate from the standard fixed summary size approach by implementing a block-wise accumulation strategy governed by a protection divisor (n). This allows us to isolate the effects of compression from sliding window artifacts. Our experiments on the Babilong benchmark reveal performance degradation in previous compression methods by 15-30% on various long context tasks. LASER-KV maintains stable performance, achieving superior accuracies by a margin of upto 10% at 128k. These findings challenge the prevailing assumption that attention scores alone are a sufficient proxy for token utility.

More Than a Quick Glance: Overcoming the Greedy Bias in KV-Cache Compression

TL;DR

The paper tackles the challenge of long-context memory under fixed KV-cache budgets in LLMs. It introduces LASER-KV, a block-wise, accumulative KV memory with a protection divisor and an Exact-LSH recall policy that fuses and LSH for token selection. On Babilong, standard compression degrades performance by about to at long contexts, while LASER-KV remains stable up to tokens and achieves up to accuracy gains at . The results argue that token utility in long contexts cannot be captured by attention scores alone and highlight the value of hybrid, structure-aware selection for sustainable long-term memory in LLMs.

Abstract

While Large Language Models (LLMs) can theoretically support extensive context windows, their actual deployment is constrained by the linear growth of Key-Value (KV) cache memory. Prevailing compression strategies mitigate this through various pruning mechanisms, yet trade-off semantic recall for memory efficiency. In this work, we present LASER-KV (Layer Accumulated Selection with Exact-LSH Recall), a framework designed to test the limits of KV compression under a strict accumulative budgeting policy. We deviate from the standard fixed summary size approach by implementing a block-wise accumulation strategy governed by a protection divisor (n). This allows us to isolate the effects of compression from sliding window artifacts. Our experiments on the Babilong benchmark reveal performance degradation in previous compression methods by 15-30% on various long context tasks. LASER-KV maintains stable performance, achieving superior accuracies by a margin of upto 10% at 128k. These findings challenge the prevailing assumption that attention scores alone are a sufficient proxy for token utility.
Paper Structure (13 sections, 1 equation, 1 figure, 3 tables, 1 algorithm)

This paper contains 13 sections, 1 equation, 1 figure, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Methodology
  4. Budget Allocation via The Protection Divisor
  5. Block Processing and Boundary Smoothing
  6. Exact-LSH Selection Algorithm
  7. Experiments
  8. Experimental Setup and Tasks
  9. Results and Analysis
  10. Conclusion
  11. Future Work
  12. Babilong Benchmark Task Descriptions
  13. Configurations

Figures (1)

  • Figure 1: The LASER-KV Selection Pipeline. The input context is divided into blocks. Our Exact-LSH mechanism then selects tokens using two criteria: Exact Attention for heavy hitters and LSH for high-recall safety, to form the compressed cache.