Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Understanding Systems Trade-offs in Retrieval-Augmented Generation Model Inference

Michael Shen, Muhammad Umar, Kiwan Maeng, G. Edward Suh, Udit Gupta

TL;DR

This paper provides a systems-oriented examination of retrieval-augmented generation (RAG) for large language models, framing a taxonomy of the RAG pipeline and building an extensible framework to study latency, throughput, and memory across offline datastore construction and online inference. Key findings show that retrieval adds substantial overhead, notably in TTFT latency and end-to-end latency, with retrieval strategy and index choice driving large accuracy–latency trade-offs. Memory and scalability emerge as critical constraints: memory-heavy indices deliver higher recall but require terabytes of memory, while memory-efficient options reduce memory use at recall costs, and datastore size can dramatically reduce throughput, motivating distributed retrieval and careful design. The work outlines concrete directions for system-level optimizations, including future exploration of sparse retrievers, query transformations, re-ranking, and end-to-end evaluation to enable practical, efficient production deployment of RAG-based LLMs.

Abstract

The rapid increase in the number of parameters in large language models (LLMs) has significantly increased the cost involved in fine-tuning and retraining LLMs, a necessity for keeping models up to date and improving accuracy. Retrieval-Augmented Generation (RAG) offers a promising approach to improving the capabilities and accuracy of LLMs without the necessity of retraining. Although RAG eliminates the need for continuous retraining to update model data, it incurs a trade-off in the form of slower model inference times. Resultingly, the use of RAG in enhancing the accuracy and capabilities of LLMs often involves diverse performance implications and trade-offs based on its design. In an effort to begin tackling and mitigating the performance penalties associated with RAG from a systems perspective, this paper introduces a detailed taxonomy and characterization of the different elements within the RAG ecosystem for LLMs that explore trade-offs within latency, throughput, and memory. Our study reveals underlying inefficiencies in RAG for systems deployment, that can result in TTFT latencies that are twice as long and unoptimized datastores that consume terabytes of storage.

Towards Understanding Systems Trade-offs in Retrieval-Augmented Generation Model Inference

TL;DR

This paper provides a systems-oriented examination of retrieval-augmented generation (RAG) for large language models, framing a taxonomy of the RAG pipeline and building an extensible framework to study latency, throughput, and memory across offline datastore construction and online inference. Key findings show that retrieval adds substantial overhead, notably in TTFT latency and end-to-end latency, with retrieval strategy and index choice driving large accuracy–latency trade-offs. Memory and scalability emerge as critical constraints: memory-heavy indices deliver higher recall but require terabytes of memory, while memory-efficient options reduce memory use at recall costs, and datastore size can dramatically reduce throughput, motivating distributed retrieval and careful design. The work outlines concrete directions for system-level optimizations, including future exploration of sparse retrievers, query transformations, re-ranking, and end-to-end evaluation to enable practical, efficient production deployment of RAG-based LLMs.

Abstract

The rapid increase in the number of parameters in large language models (LLMs) has significantly increased the cost involved in fine-tuning and retraining LLMs, a necessity for keeping models up to date and improving accuracy. Retrieval-Augmented Generation (RAG) offers a promising approach to improving the capabilities and accuracy of LLMs without the necessity of retraining. Although RAG eliminates the need for continuous retraining to update model data, it incurs a trade-off in the form of slower model inference times. Resultingly, the use of RAG in enhancing the accuracy and capabilities of LLMs often involves diverse performance implications and trade-offs based on its design. In an effort to begin tackling and mitigating the performance penalties associated with RAG from a systems perspective, this paper introduces a detailed taxonomy and characterization of the different elements within the RAG ecosystem for LLMs that explore trade-offs within latency, throughput, and memory. Our study reveals underlying inefficiencies in RAG for systems deployment, that can result in TTFT latencies that are twice as long and unoptimized datastores that consume terabytes of storage.

Paper Structure

This paper contains 7 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Retrieval-Augmented Generation
  3. Offline Datastore & Index Construction
  4. Online Inference
  5. Experimental Setup
  6. Experimental Results
  7. Conclusion

Figures (3)

  • Figure 1: Taxonomy of the online RAG inference workflow and various adjustable hyperparameters (highlighted in green), that can be used to modify and enhance the workflow.
  • Figure 2: (a) Breakdown of TTFT latency for RAG using two retrieval algorithms (HNSW-SQ and IVF-SQ) with comparable accuracy, highlighting that RAG significantly increases latency on top of prefill and decoding. (b) Analysis of RAG component tail latencies, demonstrating that unique stages of RAG, such as retrieval, experience much longer tails. (c) Examination of the continued impact of retrieval in comparison to varied encoding sequence lengths. (d) Evaluation of various stride lengths and their impacts, illustrating how aggressive strides can substantially increase the end-to-end RAG latency. We highlight a stride of 4 for optimizing perplexity seen within prior works riclm (e) Examination of the impact of varied numbers of generated tokens on all RAG components with a stride length of 4, showing a substantial increase in latency.
  • Figure 3: (a) Analysis of the trade-off between recall and latency by adjusting index-specific search-time hyperparameters, showing that higher recall often comes at the cost of lower latencies (b) Illustration of the significant increase in index storage size for datastores as the number of chunks grows, highlighting the substantial memory footprint for larger datastores. (c) Comparison of throughput versus latency for various retrieval batch sizes from 8 to 256 for, showing how HNSW indices can scale better comparatively to IVF indices. (d) Breakdown of retrieval throughput vs. latency for indices of varying datastore sizes, indicating the inability of datastores to effectively scale as larger datastores are constructed.