Trinity: Disaggregating Vector Search from Prefill-Decode Disaggregation in LLM Serving
Yi Liu, Chen Qian
TL;DR
This work addresses the challenge of integrating vector search with PD-disaggregated LLM serving to meet strict latency requirements. It proposes Trinity, a framework that routes all retrieval to a shared GPU pool for vector search, supports continuous batching on GPU ANN (e.g., CAGRA in cuVS), and uses stage-aware scheduling to balance prefill and decode SLOs. The key contributions are architectural options for vector-search placement, a continuous batching mechanism for memory-bound ANN workloads, and an adaptive two-queue scheduler that preserves both TTFT and throughput. Together, these techniques aim to improve end-to-end latency and GPU utilization in retrieval-augmented LLM serving at scale.
Abstract
Prefill and decode (PD) disaggregation separates prompt prefill and token-by-token decode stages into distinct GPU pools and has become the dominant architecture for large-scale LLM serving in industry. Also, retrieval tasks via vector search remains entangled with the model inference process, like heterogeneous RAG requests and prompt answer caches, inflating tail latency. We are motivated to investigate how vector search should be orchestrated along with PD disaggregation with a dedicated deployment architecture without violating SLOs in various retrieval workloads. We present Trinity, a practical framework that consolidates all retrieval into a single, shared vector-search GPU pool and make it work with PD disaggregated LLM serving in match. Trinity introduces (1) a novel architecture for deploying GPU-based vector search service in PD disaggregation. (2) Continuous batching for vector search that make full used of GPUs under heterogeneous queries; (3) Stage-aware scheduling that preempts vector search requests between both decode and prefill tasks.