Leveraging Approximate Caching for Faster Retrieval-Augmented Generation

TL;DR

The paper tackles the latency bottleneck of retrieval-augmented generation by introducing Proximity, an approximate key-value cache that reuses results from similar past queries to avoid expensive vector searches. It presents two variants: Proximity-FLAT, which linearly scans the entire cache, and Proximity-LSH, which uses locality-sensitive hashing to partition queries into buckets and achieve near-constant-time lookups; both are tuned with parameters such as cache capacity $c$, per-bucket size $b$, similarity threshold $ au$, and re-ranking factor $ ho$. Evaluations on MMLU and MedRAG show substantial retrieval-latency reductions (up to tens of percent to over 70% depending on workload) with only modest or no loss in end-to-end accuracy, and large-scale tests (TripClick) demonstrate robust performance and scalability. The results establish approximate similarity caching as a viable approach to accelerate RAG-based systems in realistic, skewed-query environments, with Proximity-LSH offering scalable performance through bucketed caching and SIMD-accelerated distance computations.

Abstract

Retrieval-augmented generation (RAG) improves the reliability of large language model (LLM) answers by integrating external knowledge. However, RAG increases the end-to-end inference time since looking for relevant documents from large vector databases is computationally expensive. To address this, we introduce Proximity, an approximate key-value cache that optimizes the RAG workflow by leveraging similarities in user queries. Instead of treating each query independently, Proximity reuses previously retrieved documents when similar queries appear, substantially reducing the reliance on expensive vector database lookups. To efficiently scale, Proximity employs a locality-sensitive hashing (LSH) scheme that enables fast cache lookups while preserving retrieval accuracy. We evaluate Proximity using the MMLU and MedRAG question-answering benchmarks. Our experiments demonstrate that Proximity with our LSH scheme and a realistically-skewed MedRAG workload reduces database calls by 77.2% while maintaining database recall and test accuracy. We experiment with different similarity tolerances and cache capacities, and show that the time spent within the Proximity cache remains low and constant (4.8 microseconds) even as the cache grows substantially in size. Our results demonstrate that approximate caching is a practical and effective strategy for optimizing RAG-based systems.

Figures (12)

• Figure 1: The RAG workflow.
• Figure 2: The TripClick empirical query frequencies, along with the expected frequencies from the matching Zipfian distribution.
• Figure 3: The two-dimensional projection of queries in the TripClick dataset, each one encoded using the MedCPT embedding model.
• Figure 4: The design and workflow of the Proximity approximate cache when receiving two subsequent, similar query embeddings $q_1$ and $q_2$. $q_1$ results in a cache miss whereas $q_2$ results in a hit, returning similar document indices as for $q_1$.
• Figure 5: The workflow of Proximity-LSH where each LSH bucket maps to a Proximity-FLAT cache.