From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation

TL;DR

FusionRAG addresses the latency and quality gap in retrieval-augmented generation by extending KVCache reuse into a two-stage framework that combines similarity-guided offline preprocessing with query-guided online reprocessing. It introduces practical system components—Alternative Path, asynchronous KVCache scheduling, and a sparse attention kernel (Q-Sparse-Attn)—to maintain cross-chunk context while minimizing online computation. Empirical results across multiple QA benchmarks and models show near Full Attention quality at low recomputation (around 15%), with TTFT reductions up to 9x and throughput gains up to 4x in multi-question settings. The work demonstrates a scalable, training-free approach to accelerate RAG inference while preserving fidelity to retrieved context, making it well-suited for enterprise knowledge bases and other repeatedly queried corpora.

Abstract

Retrieval-Augmented Generation enhances Large Language Models by integrating external knowledge, which reduces hallucinations but increases prompt length. This increase leads to higher computational costs and longer Time to First Token (TTFT). To mitigate this issue, existing solutions aim to reuse the preprocessed KV cache of each retrieved chunk to accelerate RAG. However, the lack of cross-chunk contextual information leads to a significant drop in generation quality, leaving the potential benefits of KV cache reuse largely unfulfilled. The challenge lies in how to reuse the precomputed KV cache of chunks while preserving generation quality. We propose FusionRAG, a novel inference framework that optimizes both the preprocessing and reprocessing stages of RAG. In the offline preprocessing stage, we embed information from other related text chunks into each chunk, while in the online reprocessing stage, we recompute the KV cache for tokens that the model focuses on. As a result, we achieve a better trade-off between generation quality and efficiency. According to our experiments, FusionRAG significantly improves generation quality at the same recomputation ratio compared to previous state-of-the-art solutions. By recomputing fewer than 15% of the tokens, FusionRAG achieves up to 70% higher normalized F1 scores than baselines and reduces TTFT by 2.66x-9.39x compared to Full Attention.

Figures (19)

• Figure 1: Comparison of Two-stage methods. Left: Existing methods (Full Reuse and CacheBlend) perform minimal offline preprocessing and rely on online recomputation. Right: FusionRAG introduces Similarity-Guided offline preprocessing (§\ref{['sec:offline_preprocessing']}) to precompute cross-attention among similar chunks, and Query-Guided selection (§\ref{['sec:online_reprocessing']}) to identify critical tokens online, achieving better quality and speed trade-offs.
• Figure 2: Comparison of the Effectiveness and Efficiency of Existing KVCache Reuse Methods. FusionRAG achieves superior performance in both generation quality and efficiency.
• Figure 3: Visualize KV deviation on a Musique question.
• Figure 4: The preprocessing and reprocessing stage in Full Reuse and CacheBlend.
• Figure 5: Full Attention is equivalent to a lower triangular mask. In contrast, Full Reuse is equivalent to Parallel Context Windows Ratner2022ParallelCW, where each token only attends to earlier tokens within the same text chunk.