Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks

TL;DR

The paper tackles the latency, error-proneness, and complexity of retrieval-augmented generation (RAG) by proposing cache-augmented generation (CAG), which preloads all relevant knowledge into a long-context LLM and caches its inference state. It encodes external knowledge into a precomputed KV cache, enabling retrieval-free inference via $r = \mathcal{M}(q \mid \mathcal{C}_{\text{KV}})$ and reducing runtime latency. Empirical evaluation on SQuAD and HotPotQA with long-context models shows CAG often matches or surpasses RAG performance while eliminating retrieval steps, especially when the knowledge base is manageable enough to fit in the extended context. The work argues for CAG as a practical, streamlined alternative to RAG for certain knowledge-intensive tasks and anticipates hybrid approaches as context windows expand.

Abstract

Retrieval-augmented generation (RAG) has gained traction as a powerful approach for enhancing language models by integrating external knowledge sources. However, RAG introduces challenges such as retrieval latency, potential errors in document selection, and increased system complexity. With the advent of large language models (LLMs) featuring significantly extended context windows, this paper proposes an alternative paradigm, cache-augmented generation (CAG) that bypasses real-time retrieval. Our method involves preloading all relevant resources, especially when the documents or knowledge for retrieval are of a limited and manageable size, into the LLM's extended context and caching its runtime parameters. During inference, the model utilizes these preloaded parameters to answer queries without additional retrieval steps. Comparative analyses reveal that CAG eliminates retrieval latency and minimizes retrieval errors while maintaining context relevance. Performance evaluations across multiple benchmarks highlight scenarios where long-context LLMs either outperform or complement traditional RAG pipelines. These findings suggest that, for certain applications, particularly those with a constrained knowledge base, CAG provide a streamlined and efficient alternative to RAG, achieving comparable or superior results with reduced complexity.

Figures (2)

• Figure 1: Comparison of Retrieval-Augmented Generation (RAG) and our Cache-Augmented Generation (CAG) Workflows: The pink-shaded components represent the processes active during real-time inference. In RAG (top section), the IR model retrieves relevant information from the knowledge source, and both the retrieved knowledge and query are processed by the LLM during inference, introducing retrieval latency. In contrast, CAG (bottom section) preloads and caches knowledge offline, allowing the LLM to process only the query during inference, eliminating retrieval overhead and ensuring a more efficient generation process.
• Figure 2: Response Time Comparison on HotPotQA (Seconds). The x-axis represents response time in seconds across different knowledge sizes. CAG eliminates retrieval overhead, while dense RAG incurs longer retrieval and generation times due to retrieving and feeding longer text chunks into the LLM. Sparse RAG retrieves shorter text spans, resulting in faster generation. As the knowledge size increases, generation time grows for all methods, but CAG remains competitive while bypassing retrieval completely.