Patchwork: A Unified Framework for RAG Serving

TL;DR

Patchwork introduces a unified end-to-end framework for deploying and serving Retrieval-Augmented Generation pipelines. By combining a lightweight Python-based RAG specification API, a profile-driven max-flow resource allocator, and an online SLO-aware scheduler, Patchwork achieves substantial throughput improvements and reduced SLO violations across heterogeneous RAG configurations. The approach preserves the user’s pipeline accuracy while dynamically adapting resources and batching to component-specific scaling patterns, enabling scalable deployment in distributed environments. Empirical results show Patchwork delivering up to ~$22 imes$ throughput gains and up to ~24% reductions in SLO violations, with modest upfront profiling costs that amortize over long online operation. This work demonstrates the feasibility and practicality of end-to-end, performance-aware RAG serving for modern, heterogeneous AI workloads.

Abstract

Retrieval Augmented Generation (RAG) has emerged as a new paradigm for enhancing Large Language Model reliability through integration with external knowledge sources. However, efficient deployment of these systems presents significant technical challenges due to their inherently heterogeneous computational pipelines comprising LLMs, databases, and specialized processing components. We introduce Patchwork, a comprehensive end-to-end RAG serving framework designed to address these efficiency bottlenecks. Patchwork's architecture offers three key innovations: First, it provides a flexible specification interface enabling users to implement custom RAG pipelines. Secondly, it deploys these pipelines as distributed inference systems while optimizing for the unique scalability characteristics of individual RAG components. Third, Patchwork incorporates an online scheduling mechanism that continuously monitors request load and execution progress, dynamically minimizing SLO violations through strategic request prioritization and resource auto-scaling. Our experimental evaluation across four distinct RAG implementations demonstrates that Patchwork delivers substantial performance improvements over commercial alternatives, achieving throughput gains exceeding 48% while simultaneously reducing SLO violations by ~24%.

Figures (18)

• Figure 1: Overview of RAG Pipeline: The above figure shows a schematic of RAG pipeline with details in Section 2. RAG logic can vary dramatically from simple concatenation operation to a complicated execution logic with conditional and recursive operations.
• Figure 2: Components of a RAG pipeline: The above provides a schematic of RAG execution pipeline. The box with solid colors represent the mandatory components, while the the box with hatch pattern represents the optional pattern. The text below the boxes represent a subset of parameters on which the scalability depends.
• Figure 3: Scaling with documents: In the above figure, we show the time to find five nearest neighbors using L2 distance metrics scale with number of documents for two popular vector databases- Qdrant and ChromaDB. We observe that the time taken per query by Qdrant scales faster than ChromaDB, highlighting different scalability patterns.
• Figure 4: Scaling with candidates: In the above figure, we show how the time for ChromaDB and Qdrant scales with different numbers of nearest neighbors. We observe that these two databases behave differently: ChromaDB remains largely constant as the number of nearest neighbors increases, while Qdrant scales at a high rate.
• Figure 5: Scaling behavior of Qdrant and ChromaDB with batch size, using cosine similarity as the distance metric.