Taming the Titans: A Survey of Efficient LLM Inference Serving

TL;DR

The paper investigates the memory and compute bottlenecks of LLM inference serving, with a focus on attention-driven overhead and latency targets. It presents a hierarchical taxonomy—from instance-level optimizations (model placement, scheduling, decoding-length prediction, KV cache, and disaggregation) to cluster-level strategies (heterogeneous resource optimization, load balancing, and cloud-edge deployment)—and covers emerging scenarios (long context, RAG, MoE, LoRA, speculative decoding, augmented LLMs, and test-time reasoning) as well as miscellaneous areas (hardware, privacy, simulators, fairness, energy). Key contributions include a fine-grained taxonomy, synthesis of representative methods, and forward-looking research directions to enhance SLO-compliant LLM serving at scale. The analysis highlights how techniques such as $O(n^2)$ attention complexity, $O(n)$ decoding with KV cache, disaggregation, and intelligent scheduling collectively enable practical, low-latency LLM services across on-premise, cloud, and edge environments. The work provides actionable guidance for practitioners and researchers aiming to deploy and optimize large-scale inference systems with tight latency and cost constraints.

Abstract

Large Language Models (LLMs) for Generative AI have achieved remarkable progress, evolving into sophisticated and versatile tools widely adopted across various domains and applications. However, the substantial memory overhead caused by their vast number of parameters, combined with the high computational demands of the attention mechanism, poses significant challenges in achieving low latency and high throughput for LLM inference services. Recent advancements, driven by groundbreaking research, have significantly accelerated progress in this field. This paper provides a comprehensive survey of these methods, covering fundamental instance-level approaches, in-depth cluster-level strategies, emerging scenario directions, and other miscellaneous but important areas. At the instance level, we review model placement, request scheduling, decoding length prediction, storage management, and the disaggregation paradigm. At the cluster level, we explore GPU cluster deployment, multi-instance load balancing, and cloud service solutions. For emerging scenarios, we organize the discussion around specific tasks, modules, and auxiliary methods. To ensure a holistic overview, we also highlight several niche yet critical areas. Finally, we outline potential research directions to further advance the field of LLM inference serving.

Figures (7)

• Figure 1: Overview of the paper, detailing Instance, Cluster, Emerging Scenarios, and Miscellaneous Areas. $\mathbf{R}$ represents a request. In Inter-request scheduling, two requests, $\mathbf{R1}$ (10 toks) and $\mathbf{R2}$ (2 toks), arrive simultaneously. Ignoring the prefill process, if $\mathbf{R1}$ is processed first, its generation rate is 1 tok/s, and $\mathbf{R2}$'s rate is 0.2 tok/s. Reversing the order gives $\mathbf{R2}$ a rate of 1 tok/s and $\mathbf{R1}$ 0.9 tok/s. The default decoding speed is 1 token/s.
• Figure 2: Illustration of the LLM Inference process.
• Figure 3: Taxonomy of Instance-Level approaches for LLM inference serving.
• Figure 4: Taxonomy of Cluster-Level strategies for LLM inference serving.
• Figure 5: Taxonomy of Emerging Scenarios for LLM inference serving.