CaGR-RAG: Context-aware Query Grouping for Disk-based Vector Search in RAG Systems

TL;DR

This work targets inefficiencies in disk-based vector search for Retrieval Augmented Generation (RAG) systems by addressing non-uniform cluster access patterns of queries. It introduces CaGR-RAG, a context-aware query grouping mechanism that uses the Jaccard similarity $J(q_i,q_j) = \frac{|C(q_i)\cap C(q_j)|}{|C(q_i)\cup C(q_j)|}$ with a threshold $\theta$ to cluster queries by shared cluster access and an opportunistic prefetch module to preload the next group’s clusters. Evaluations on BEIR-derived datasets with IVF FAISS show substantial improvements in tail latency (up to 51.55% at the 99th percentile) and higher, more stable cache hit ratios compared with the EdgeRAG baseline. The approach offers practical gains in retrieval latency for large-scale RAG deployments and is compatible with existing cache replacement strategies, providing a principled way to exploit query context for disk-based vector search.

Abstract

Modern embedding models capture both semantic and syntactic structures of queries, often mapping different queries to similar regions in vector space. This results in non-uniform cluster access patterns in disk-based vector search systems, particularly in Retrieval Augmented Generation (RAG) framework. While existing approaches optimize individual queries, they overlook the impact of cluster access patterns, failing to account for the locality effects of queries that access similar clusters. This oversight reduces cache efficiency and increases search latency due to excessive disk I/O. To address this, we introduce CaGR-RAG, a context-aware query grouping mechanism that organizes queries based on shared cluster access patterns. Additionally, it incorporates opportunistic cluster prefetching to minimize cache misses during transitions between query groups, further optimizing retrieval performance. Experimental results show that CaGR-RAG reduces 99th percentile tail latency by up to 51.55% while consistently maintaining a higher cache hit ratio than the baseline.

Figures (7)

• Figure 1: Cluster accessed pattern per an embedding model.
• Figure 2: The cumulative distribution function of search latency per nprobe and the relation between cache hit ratio and search latency when nprobe is set to 40.
• Figure 3: Illustration of disk-based IVF search for the baseline and CaGR-RAG, respectively. In the illustration, we explain vector search steps of query 2 and 3. We assume that number of total cache entries is 5. Initially, four queries arrive.
• Figure 4: Cache utilization of EdgeRAG and CaGR-RAG under three datasets.
• Figure 5: The relationship between file size reading from disk, search latency, and cache hit ratio in hotpotqa dataset. Note that (a) and (b) use a different scale on the right y-axis.