Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SABlock: Semantic-Aware KV Cache Eviction with Adaptive Compression Block Size

Jinhan Chen, Jianchun Liu, Hongli Xu, Xianjun Gao, Shilong Wang

TL;DR

KV caches in long-context LLM inference impose severe memory demands that scale with sequence length. SABlock tackles this by combining semantic segmentation, segment-guided token scoring, and a budget-driven adaptive block-size search to preserve critical, semantically coherent content under fixed memory budgets. Empirical results on LongBench and Needle-in-a-Haystack show SABlock substantially reduces peak memory (e.g., 46.28% at 128K context) and accelerates decoding (up to 9.5x) while maintaining near-full accuracy (e.g., 99.9% retrieval with 96 KV entries). The approach enables practical long-context inference across models, offering a content-aware, memory-efficient KV-cache eviction paradigm with strong performance and robustness across budgets.

Abstract

The growing memory footprint of the Key-Value (KV) cache poses a severe scalability bottleneck for long-context Large Language Model (LLM) inference. While KV cache eviction has emerged as an effective solution by discarding less critical tokens, existing token-, block-, and sentence-level compression methods struggle to balance semantic coherence and memory efficiency. To this end, we introduce SABlock, a \underline{s}emantic-aware KV cache eviction framework with \underline{a}daptive \underline{block} sizes. Specifically, SABlock first performs semantic segmentation to align compression boundaries with linguistic structures, then applies segment-guided token scoring to refine token importance estimation. Finally, for each segment, a budget-driven search strategy adaptively determines the optimal block size that preserves semantic integrity while improving compression efficiency under a given cache budget. Extensive experiments on long-context benchmarks demonstrate that SABlock consistently outperforms state-of-the-art baselines under the same memory budgets. For instance, on Needle-in-a-Haystack (NIAH), SABlock achieves 99.9% retrieval accuracy with only 96 KV entries, nearly matching the performance of the full-cache baseline that retains up to 8K entries. Under a fixed cache budget of 1,024, SABlock further reduces peak memory usage by 46.28% and achieves up to 9.5x faster decoding on a 128K context length.

SABlock: Semantic-Aware KV Cache Eviction with Adaptive Compression Block Size

TL;DR

KV caches in long-context LLM inference impose severe memory demands that scale with sequence length. SABlock tackles this by combining semantic segmentation, segment-guided token scoring, and a budget-driven adaptive block-size search to preserve critical, semantically coherent content under fixed memory budgets. Empirical results on LongBench and Needle-in-a-Haystack show SABlock substantially reduces peak memory (e.g., 46.28% at 128K context) and accelerates decoding (up to 9.5x) while maintaining near-full accuracy (e.g., 99.9% retrieval with 96 KV entries). The approach enables practical long-context inference across models, offering a content-aware, memory-efficient KV-cache eviction paradigm with strong performance and robustness across budgets.

Abstract

The growing memory footprint of the Key-Value (KV) cache poses a severe scalability bottleneck for long-context Large Language Model (LLM) inference. While KV cache eviction has emerged as an effective solution by discarding less critical tokens, existing token-, block-, and sentence-level compression methods struggle to balance semantic coherence and memory efficiency. To this end, we introduce SABlock, a \underline{s}emantic-aware KV cache eviction framework with \underline{a}daptive \underline{block} sizes. Specifically, SABlock first performs semantic segmentation to align compression boundaries with linguistic structures, then applies segment-guided token scoring to refine token importance estimation. Finally, for each segment, a budget-driven search strategy adaptively determines the optimal block size that preserves semantic integrity while improving compression efficiency under a given cache budget. Extensive experiments on long-context benchmarks demonstrate that SABlock consistently outperforms state-of-the-art baselines under the same memory budgets. For instance, on Needle-in-a-Haystack (NIAH), SABlock achieves 99.9% retrieval accuracy with only 96 KV entries, nearly matching the performance of the full-cache baseline that retains up to 8K entries. Under a fixed cache budget of 1,024, SABlock further reduces peak memory usage by 46.28% and achieves up to 9.5x faster decoding on a 128K context length.
Paper Structure (18 sections, 13 equations, 10 figures, 4 tables, 2 algorithms)

This paper contains 18 sections, 13 equations, 10 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. KV Cache in LLM Inference
  4. Limitations of Existing KV Cache Eviction Methods
  5. Opportunity for Adaptive Block Size Configuration
  6. Framework Design
  7. Semantic-Aware Segmentation
  8. Segment-Guided Token Scoring
  9. Budget-Driven Adaptive Block Size Search
  10. Performance Evaluation
  11. Experimental Settings
  12. Overall Performance on LongBench
  13. Context Retrieval under NIAH
  14. Evaluation of Resource Consumption
  15. Ablation Studies
  16. ...and 3 more sections

Figures (10)

  • Figure 1: Segmentation strategies applied to the same prompt: token-level, block-level, sentence-level, and our semantic-aware adaptive blocks (the first segment uses a block size of 3 and the second segment uses a block size of 2). Red tokens indicate evicted content, while bold tokens denote preserved content.
  • Figure 2: Illustration of the limitations of block-level KV cache eviction strategies.
  • Figure 2: Detailed accuracy comparison on 16 LongBench tasks with different KV cache budgets. The optimal result is shown in bold, and the sub-optimal result is underlined.
  • Figure 3: Performance comparison under a fixed KV cache budget of 96 on three categories of the LongBench benchmark. We compare SentenceKV, SentenceKV with an additional block size of 5 (SentenceKV w. Block(5)), and our method.
  • Figure 4: Answer accuracy under different cache budgets and compression block sizes. The first example demonstrates that compression with a small block size better retains critical information under a low budget, while the second shows that our adaptive method succeeds when fixed block sizes fail.
  • ...and 5 more figures