RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving

TL;DR

This work addresses the challenge of efficiently serving retrieval-augmented generation (RAG) systems amid heterogeneous workloads. It introduces RAGSchema, a workload abstraction that captures RAG pipelines and configurations, and RAGO, a systematic optimizer that searches for Pareto-optimal scheduling policies across placement, allocation, and batching. Through four representative paradigms, the authors demonstrate substantial performance variability and bottlenecks, motivating a formal optimization framework. Empirical results show RAGO delivers up to 2× QPS per chip and a 55% reduction in time-to-first-token latency compared with strong LLM-only baselines, underscoring the practical impact of workload-aware RAG optimization for deployment at scale.

Abstract

Retrieval-augmented generation (RAG), which combines large language models (LLMs) with retrievals from external knowledge databases, is emerging as a popular approach for reliable LLM serving. However, efficient RAG serving remains an open challenge due to the rapid emergence of many RAG variants and the substantial differences in workload characteristics across them. In this paper, we make three fundamental contributions to advancing RAG serving. First, we introduce RAGSchema, a structured abstraction that captures the wide range of RAG algorithms, serving as a foundation for performance optimization. Second, we analyze several representative RAG workloads with distinct RAGSchema, revealing significant performance variability across these workloads. Third, to address this variability and meet diverse performance requirements, we propose RAGO (Retrieval-Augmented Generation Optimizer), a system optimization framework for efficient RAG serving. Our evaluation shows that RAGO achieves up to a 2x increase in QPS per chip and a 55% reduction in time-to-first-token latency compared to RAG systems built on LLM-system extensions.

Figures (19)

• Figure 1: LLM-only system (top) versus RAG (bottom).
• Figure 2: RAGO for systematic RAG serving optimization.
• Figure 3: Describing general RAG pipelines with RAGSchema.
• Figure 4: Parallelisms and cost model in model inference.
• Figure 5: Larger LLM versus RAG with smaller models.