An experimental study of KV cache reuse strategies in chunk-level caching systems

Abstract

Retrieval-augmented generation improves large language models' accuracy by adding relevant retrieved text to the prompt. Chunk level caching (CLC) accelerates inference by precomputing KV caches for these retrieved chunks and reusing them. However, these caches miss cross-attention dependencies between chunks, which can reduce output quality. Several methods try to improve CLC accuracy using different techniques. We make two main contributions. First, we show that existing CLC approaches have fundamental limitations that limit their accuracy or their applicability. We back this conclusion with an extensive CLC system experimental evaluation. Second, we observe that existing CLC techniques are complementary. We leverage this insight to propose a new CLC design that carefully combines them and achieves better accuracy.

Figures (9)

• Figure 1: Top: KV cache and attention profile with full-prefill. Middle: CLC with naive KV cache reuse lacks cross-chunk attention, and obtains different attention profiles. Bottom: selective recomputation aims to recover key cross-chunk attention information; attention reshaping aims to obtain an attention profile similar to full prefill's.
• Figure 2: Cacheblend accuracy.
• Figure 3: Ideal token selection using $\Delta K$ ($r=15\%$) for one query in Llama-2WikiMQA (left) and Qwen-Musique (right). Black/white: a token is/is not selected at that layer. The clusters of chosen tokens at early layers correspond to beginning-of-chunk tokens.
• Figure 4: Left: EPIC/Link0 accuracy. Center: cumulative percentage of tokens' $\Delta K$ for Llama on 2WikiMQA and Qwen on Musique, with and without the first 10 tokens (sink) of each chunk. Right: CacheClip accuracy.
• Figure 5: Average percentage of the tokens chosen (by $\Delta K$) for recomputation at a reference layer that would be chosen also in other layers ($r=15\%$).