SpotServe: Serving Generative Large Language Models on Preemptible Instances

TL;DR

The paper tackles the high cost of serving generative LLMs by leveraging preemptible cloud GPUs. SpotServe enables distributed LLM inference across spot instances by dynamic reparallelization, efficient context migration via a KM-based device mapping, and stateful inference recovery leveraging cloud grace periods. It introduces a parallelization controller, a device mapper, and a migration planner along with a just-in-time arrangement strategy. Evaluation on real traces and multiple models shows up to 2.4-9.1x P99 latency improvement and up to 54% cost savings versus on-demand, demonstrating practical cost-effective LLM serving.

Abstract

The high computational and memory requirements of generative large language models (LLMs) make it challenging to serve them cheaply. This paper aims to reduce the monetary cost for serving LLMs by leveraging preemptible GPU instances on modern clouds, which offer accesses to spare GPUs at a much cheaper price than regular instances but may be preempted by the cloud at any time. Serving LLMs on preemptible instances requires addressing challenges induced by frequent instance preemptions and the necessity of migrating instances to handle these preemptions. This paper presents SpotServe, the first distributed LLM serving system on preemptible instances. Several key techniques in SpotServe realize fast and reliable serving of generative LLMs on cheap preemptible instances. First, SpotServe dynamically adapts the LLM parallelization configuration for dynamic instance availability and fluctuating workload, while balancing the trade-off among the overall throughput, inference latency and monetary costs. Second, to minimize the cost of migrating instances for dynamic reparallelization, the task of migrating instances is formulated as a bipartite graph matching problem, which uses the Kuhn-Munkres algorithm to identify an optimal migration plan that minimizes communications. Finally, to take advantage of the grace period offered by modern clouds, we introduce stateful inference recovery, a new inference mechanism that commits inference progress at a much finer granularity and allows SpotServe to cheaply resume inference upon preemption. We evaluate on real spot instance preemption traces and various popular LLMs and show that SpotServe can reduce the P99 tail latency by 2.4 - 9.1x compared with the best existing LLM serving systems. We also show that SpotServe can leverage the price advantage of preemptive instances, saving 54% monetary cost compared with only using on-demand instances.

Figures (9)

• Figure 1: Illustration of incremental decoding in generative LLM and distributed LLM inference on preemptible instances
• Figure 2: Illustration of different model parallelisms
• Figure 3: An overview of SpotServe.
• Figure 4: Figure \ref{['fig:reparallel']} shows an example of SpotServe changes the parallel configuration from (1,2,8) to (1,3,4) through context migration within the grace period and continues previous decoding progress of request $r_3$. Figure \ref{['fig:mapping']} shows an example bipartite graph between six available instances (i.e., $u_0\sim u_5$) and topology positions in the new configuration (2,3,1). Here we only draw the weighted edges starting from $u_1$.
• Figure 5: Trace $A_S$ and $B_S$ are extracted from real trace, while $A_{S+O}$ and $B_{S+O}$ are traces created by Algorithm \ref{['algo:opt']} mixing on-demand instances based on $A_S$ and $B_S$. Each instance has four GPUs.