Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SeaLLM: Service-Aware and Latency-Optimized Resource Sharing for Large Language Model Inference

Yihao Zhao, Jiadun Chen, Peng Sun, Lei Li, Xuanzhe Liu, Xin Jin

TL;DR

SeaLLM tackles inefficient GPU utilization in multi-LLM inference by enabling service-aware sharing across diverse LLMs. It introduces a latency-optimized scheduling policy, a two-stage placement with adaptive replacement, and a unified KV cache to enable efficient cross-LLM memory sharing. Evaluations on real traces and a 32-GPU cluster show substantial reductions in normalized and tail latency and improved SLO attainment compared to state-of-the-art baselines, with notable token-level gains as well. The work demonstrates practical impact for cloud LLM services by improving resource utilization and responsiveness under dynamic workloads.

Abstract

Large language models (LLMs) with different architectures and sizes have been developed. Serving each LLM with dedicated GPUs leads to resource waste and service inefficiency due to the varying demand of LLM requests. A common practice is to share multiple LLMs. However, existing sharing systems either do not consider the autoregressive pattern of LLM services, or only focus on improving the throughput, which impairs the sharing performance, especially the serving latency. We present SeaLLM, which enables service-aware and latency-optimized LLM sharing. SeaLLM improves the overall sharing performance by (1) a latency-optimized scheduling algorithm utilizing the characteristics of LLM services, (2) a placement algorithm to determine the placement plan and an adaptive replacement algorithm to decide the replacement interval, and (3) a unified key-value cache to share GPU memory among LLM services efficiently. Our evaluation under real-world traces and LLM services demonstrates that SeaLLM improves the normalized latency by up to $13.60\times$, the tail latency by up to $18.69\times$, and the SLO attainment by up to $3.64\times$ compared to existing solutions.

SeaLLM: Service-Aware and Latency-Optimized Resource Sharing for Large Language Model Inference

TL;DR

SeaLLM tackles inefficient GPU utilization in multi-LLM inference by enabling service-aware sharing across diverse LLMs. It introduces a latency-optimized scheduling policy, a two-stage placement with adaptive replacement, and a unified KV cache to enable efficient cross-LLM memory sharing. Evaluations on real traces and a 32-GPU cluster show substantial reductions in normalized and tail latency and improved SLO attainment compared to state-of-the-art baselines, with notable token-level gains as well. The work demonstrates practical impact for cloud LLM services by improving resource utilization and responsiveness under dynamic workloads.

Abstract

Large language models (LLMs) with different architectures and sizes have been developed. Serving each LLM with dedicated GPUs leads to resource waste and service inefficiency due to the varying demand of LLM requests. A common practice is to share multiple LLMs. However, existing sharing systems either do not consider the autoregressive pattern of LLM services, or only focus on improving the throughput, which impairs the sharing performance, especially the serving latency. We present SeaLLM, which enables service-aware and latency-optimized LLM sharing. SeaLLM improves the overall sharing performance by (1) a latency-optimized scheduling algorithm utilizing the characteristics of LLM services, (2) a placement algorithm to determine the placement plan and an adaptive replacement algorithm to decide the replacement interval, and (3) a unified key-value cache to share GPU memory among LLM services efficiently. Our evaluation under real-world traces and LLM services demonstrates that SeaLLM improves the normalized latency by up to , the tail latency by up to , and the SLO attainment by up to compared to existing solutions.

Paper Structure

This paper contains 22 sections, 3 theorems, 13 equations, 15 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. LLM Service
  4. LLM Sharing Systems
  5. Challenges
  6. SeaLLM Overview
  7. Design
  8. Service Placement
  9. Latency-Optimized Scheduling
  10. Adaptive Replacement
  11. Unified KV Cache
  12. Implementation
  13. Evaluation
  14. Experiment Setup
  15. End-to-End Performance
  16. ...and 7 more sections

Key Result

Theorem 1

The proposed scheduling algorithm can minimize the normalized latency with the following assumptions: (1) requests can be preempted at any time, and (2) all requests of the same service have identical execution time.

Figures (15)

  • Figure 1: The request traffic of four production LLM services during seven days stojkovic2024dynamollmwang2024towards.
  • Figure 2: Four execution strategies of serving two LLM services on two GPUs. The blocks with dark colour are of the prefill phase and the blocks with light colour are of the decoding phase. $L_0$ and $L_1$ represent the average latency of service $S_0$ and $S_1$, respectively. $L_N$ represents the normalized latency of $S_0$ and $S_1$, defined in Equation \ref{['equ:norm_latency']}.
  • Figure 3: The iteration time of the prefill phase and the decoding phase under different TP sizes. Len represents the number of input tokens and the batch size is 8.
  • Figure 4: SeaLLM architecture.
  • Figure 5: Comparison among (a) single LLM's cache block, (b) split cache block, and (c) merged cache block.
  • ...and 10 more figures

Theorems & Definitions (5)

  • Theorem 1
  • Theorem 2
  • proof
  • proof
  • Lemma 1