Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving

TL;DR

Apt-Serve addresses the TTFT SLO bottleneck in scalable LLM inference serving by coupling a hybrid KV/hidden cache with adaptive per-iteration scheduling. The scheduling problem is formalized as a per-iteration optimization that is NP-hard, and a greedy $2$-approximation guarantees practical performance. Empirical results on three real-world datasets and LLMs from $13$B to $66$B parameters show up to $8.8\times$ improvement in effective throughput over state-of-the-art systems, with strong robustness under bursty traffic and across diverse prompts. The approach enables larger batch sizes within memory constraints and dynamic batch composition, offering a practical pathway to real-time, scalable LLM services.

Abstract

Large language model (LLM) inference serving systems are essential to various LLM-based applications. As demand for LLM services continues to grow, scaling these systems to handle high request rates while meeting latency Service-Level Objectives (SLOs), referred to as effective throughput, becomes critical. However, existing systems often struggle to improve effective throughput, primarily due to a significant decline in Time To First Token (TTFT) SLO attainment. We identify two major causes of this bottleneck: (1) memory-intensive KV cache that limits batch size expansion under GPU memory constraints, and (2) rigid batch composition enforced by the default First-Come-First-Serve scheduling policy. In this paper, we introduce Apt-Serve, a scalable framework designed to enhance effective throughput in LLM inference serving. Apt-Serve features a new hybrid cache scheme that combines KV cache with a memory-efficient hidden cache for reusable input hidden state vectors, allowing large batch sizes and improving request concurrency. Based on the hybrid cache, Apt-Serve employs an adaptive runtime scheduling mechanism that dynamically optimizes batch composition. We formally define the adaptive scheduling optimization problem and propose an efficient algorithm with theoretical guarantees. Extensive evaluations on three real-world datasets and LLMs ranging from 13B to 66B parameters demonstrate that Apt-Serve achieves up to 8.8x improvement in effective throughput compared to the state-of-the-art inference serving systems.

Figures (12)

• Figure 1: Serving sampled requests from the ShareGPT sharegpt dataset using the vLLM kwon2023efficient system with the OPT-13B model zhang2022opt on an NVIDIA A100 GPU. The X-axis represents the request rate (req/s), and the Y-axis shows the SLO attainment (%) (TTFT: 1s, P99 TBT: 1s).
• Figure 2: (a) SLO attainment rate (%) and time ratio at batch size limit (%) under varying request rates. The left Y-axis is the percentage of served requests adhering to the SLOs, while the right Y-axis is the percentage of serving time the system operates at the batch size limit. The X-axis is the varied request rates. (b) A comparison of specific SLO attainments at request rates of 2.0/reqs and 3.0 req/s.
• Figure 3: The illustrations of the computations performed during the decode phase for a request (a) with KV cache and (b) with hidden cache. Assume the original input hidden state vectors for the target request consist of a length of 3 (1 current token + 2 past tokens), and each vector has a dimension of 3.
• Figure 4: (a) SLO attainment (%) comparison between FCFS and random scheduling policies. The X-axis represents the request rate, and the Y-axis shows the percentage of requests meeting SLOs. (b) Distribution of per-request SLO attainment using FCFS scheduling. (c) Distribution of per-request SLO attainment using random scheduling. In (b) and (c), the X-axis displays request IDs sorted by arrival time, the upper Y-axis is the TTFT latency, and the lower Y-axis is the P99 TBT latency for served requests.
• Figure 5: An overview of Apt-Serve.

Theorems & Definitions (1)

• Definition 1: Hybrid-cache-based Scheduling Problem