VectorLiteRAG: Latency-Aware and Fine-Grained Resource Partitioning for Efficient RAG

TL;DR

VectorLiteRAG addresses the bottleneck of co-locating vector retrieval and LLM inference by introducing a latency-aware, fine-grained CPU-GPU index partitioning strategy for IVF-based retrieval. It combines profiling-based performance modeling, tail hit-rate estimation via a Beta distribution, and a latency-bounded partitioning algorithm to place hot clusters on GPUs while keeping CQ on the CPU, supported by a distributed runtime with a Router, Dynamic Dispatcher, and adaptive index updates. The approach yields up to 1.5x higher SLO-attainable throughput across varied LLMs and vector databases, with robust performance under varying input/output lengths and hardware capacities. Practically, VectorLiteRAG enables scalable, real-time RAG serving without extra hardware, by balancing resource contention and exploiting inter-device hit-rate variance to sustain end-to-end latency targets.

Abstract

Retrieval-Augmented Generation (RAG) systems combine vector similarity search with large language models (LLMs) to deliver accurate, context-aware responses. However, co-locating the vector retriever and the LLM on shared GPU infrastructure introduces significant challenges: vector search is memory and I/O intensive, while LLM inference demands high throughput and low latency. Naive resource sharing often leads to severe performance degradation, particularly under high request load or large index sizes. We present VectorLiteRAG, a deployment-friendly RAG system that achieves latency-compliant inference without requiring additional hardware resources. VectorLiteRAG introduces a fine-grained GPU resource allocation mechanism based on detailed performance modeling and access pattern analysis. By estimating search latency and query hit rate distributions, it identifies an optimal index partitioning point across CPU and GPU tiers to minimize contention and maximize throughput. Our evaluations show that VectorLiteRAG consistently expands the SLO compliant request rate range across all tested configurations, including both small and large LLMs, and small and large vector databases compared to naive baselines and state of the art alternatives. In the best case, VectorLiteRAG improves the attainable SLO throughput by up to 1.5 times without compromising generation quality or requiring additional compute resources.

Figures (17)

• Figure 1: End-to-end pipeline of a RAG system, where the input query is indexed into the vector database stored in memory, while the knowledge corpus resides in storage. The LLM prefill and decode execute on the GPU.
• Figure 2: Three stages of vector search in IVF–based index: (1) coarse quantization to identify clusters most semantically similar to the query, (2) construction of a LUT containing partial distances between the query and codewords, and (3) scanning the LUT and re-ranking candidates from the selected clusters based on aggregated distances.
• Figure 3: Left: Search latency comparison between standard IVF and IVF with fast scan (IVF-FS). Except for the fast scan optimization, both indexes share identical configurations. IVF-FS achieves significantly faster search speed. Right: Latency breakdown of IVF-FS on a 128M vector index. Lookup table operations dominate the overall search time.
• Figure 4: Left: While fast scanning accelerates IVF-based vector search on CPU(64 core Xeon 8462Y+), GPU(H100)-based IVF search offers superior performance. Right: Relationship between KV cache size and LLM throughput for the Qwen3-30B model on two H100 GPUs. Reducing KV cache space leads to a significant drop in throughput.
• Figure 5: CDF of cluster access frequency for queries from the Wiki-All bib:wikiall and ORCAS bib:orcas datasets. While the two distributions exhibit different levels of skewness, in both cases, the top 20% of clusters account for over 50% of the total distance computations.