DynSplit-KV: Dynamic Semantic Splitting for KVCache Compression in Efficient Long-Context LLM Inference

TL;DR

Long-context KVCache growth creates memory and latency bottlenecks in LLM inference. The paper introduces DynSplit-KV, which combines dynamic delimiter selection (DD-Select) with a variable-to-fixed block mapping (V2F) to compress KVCache without sacrificing semantic fidelity. By scoring delimiters with attention-based importance and translating variable-length semantic blocks into fixed-length representations, the approach achieves substantially higher accuracy and significant efficiency gains, including up to $2.2\times$ speedups compared with FlashAttention and $2.6\times$ peak memory reduction on long-context benchmarks. Evaluations across LongBench, LongBench v2, and passkey tasks on multiple models demonstrate robust performance improvements and scalability, including CPU-GPU offloading scenarios for ultra-long contexts.

Abstract

Although Key-Value (KV) Cache is essential for efficient large language models (LLMs) inference, its growing memory footprint in long-context scenarios poses a significant bottleneck, making KVCache compression crucial. Current compression methods rely on rigid splitting strategies, such as fixed intervals or pre-defined delimiters. We observe that rigid splitting suffers from significant accuracy degradation (ranging from 5.5% to 55.1%) across different scenarios, owing to the scenario-dependent nature of the semantic boundaries. This highlights the necessity of dynamic semantic splitting to match semantics. To achieve this, we face two challenges. (1) Improper delimiter selection misaligns semantics with the KVCache, resulting in 28.6% accuracy loss. (2) Variable-length blocks after splitting introduce over 73.1% additional inference overhead. To address the above challenges, we propose DynSplit-KV, a KVCache compression method that dynamically identifies delimiters for splitting. We propose: (1) a dynamic importance-aware delimiter selection strategy, improving accuracy by 49.9%. (2) A uniform mapping strategy that transforms variable-length semantic blocks into a fixed-length format, reducing inference overhead by 4.9x. Experiments show that DynSplit-KV achieves the highest accuracy, 2.2x speedup compared with FlashAttention and 2.6x peak memory reduction in long-context scenarios.

Figures (12)

• Figure 2: Challenges, solutions, and performance summary. Note: Accuracy data are obtained from experimental evaluations of baselines and our method on datasets of LongBench bai2023longbench.
• Figure 3: Observation. (a) Impacts of rigid splitting and delimiter importance (across (b) scenarios/ (c) models, (d) on accuracy).
• Figure 4: DD-Select Pipeline.
• Figure 5: (a) Compression Strategy. (b) Variable-to-Fixed Block Mapping Strategy.
• Figure 6: Accuracy trends on LongBench under different KVCache usage.