TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval

TL;DR

TeleRAG tackles the latency and memory bottlenecks of retrieval-augmented generation by introducing lookahead retrieval, which prefetches likely-needed IVF clusters to the GPU during LLM generation to hide CPU–GPU transfer costs. It combines a prefetching scheduler and a cache-aware scheduling strategy to enable efficient batched and multi-GPU inference with minimal GPU memory overhead. Empirical results across six RAG pipelines and multiple hardware setups show up to $1.53\times$ end-to-end latency reduction on single queries and up to $1.83\times$ throughput gains in batched runs, with strong scaling on multi-GPU systems (e.g., up to $6.5\times$ on 8 GPUs). The work demonstrates that memory-efficient GPU-accelerated retrieval, coupled with intelligent scheduling and caching, can deliver practical, high-throughput RAG deployments for large-scale datastores.

Abstract

Retrieval-augmented generation (RAG) extends large language models (LLMs) with external data sources to enhance factual correctness and domain coverage. Modern RAG pipelines rely on large datastores, creating a significant system challenge: achieving high throughput and low latency is difficult, especially when GPU memory is limited. To address these challenges, we propose TeleRAG, an efficient inference system that reduces latency and improves throughput with minimal GPU memory requirements. The core innovation of TeleRAG is lookahead retrieval, a prefetching mechanism that predicts required data and transfers them from CPU to GPU in parallel with LLM generation. In addition, TeleRAG adopts a prefetching scheduler and a cache-aware scheduler to support efficient multi-GPU inference with minimal overhead. Evaluations show TeleRAG achieves up to a 1.53x average end-to-end latency reduction (single-query) and 1.83x higher average throughput (batched), as well as good scalability in throughput. This confirms the practical utility of TeleRAG for faster and more memory-efficient deployments of RAG applications.

Figures (15)

• Figure 1: Typical pipeline stages of a RAG application.
• Figure 2: Illustration of TeleRAG's lookahead retrieval mechanism and comparison to different RAG systems. TeleRAG prefetches relevant retrieval data from CPU to GPU, overlaps data transfer with the pre-retrieval stage, and accelerates retrieval with GPU--CPU cooperation.
• Figure 3: Overview of RAG.
• Figure 4: Latency breakdown of six RAG pipelines on NQ dataset kwiatkowski2019natural with one H100 GPU.
• Figure 5: Latency breakdown of CPU-offload and runtime-fetch GPU retrieval, averaged over 512 random NQ queries.