Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism

Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, Xin Jin

TL;DR

LoongServe introduces Elastic Sequence Parallelism (ESP) to dynamically adjust parallelism degree across iterations for long-context LLM serving, addressing variability in compute and memory demands. It implements zero-overhead elastic scale-down via proactive KV-cache migration during prefill and multi-master distributed decoding for scale-up, coupled with a four-step, real-time scheduling algorithm that optimizes dispatching, allocation, batching, and scaling plans. A unified distributed KV cache pool and carefully engineered routing and kernel optimizations enable efficient resource sharing and reduced memory fragmentation. Evaluations on long-context models and diverse workloads show up to 3.85× throughput gains over chunked prefill and 5.81× over prefill-decoding disaggregation, with robust multi-node scalability and strong analytical modeling for planning.

Abstract

The context window of large language models (LLMs) is rapidly increasing, leading to a huge variance in resource usage between different requests as well as between different phases of the same request. Restricted by static parallelism strategies, existing LLM serving systems cannot efficiently utilize the underlying resources to serve variable-length requests in different phases. To address this problem, we propose a new parallelism paradigm, elastic sequence parallelism (ESP), to elastically adapt to the variance between different requests and phases. Based on ESP, we design and build LoongServe, an LLM serving system that (1) improves computation efficiency by elastically adjusting the degree of parallelism in real-time, (2) improves communication efficiency by reducing key-value cache migration overhead and overlapping partial decoding communication with computation, and (3) improves GPU memory efficiency by reducing key-value cache fragmentation across instances. Our evaluation under diverse real-world datasets shows that LoongServe improves the maximum throughput by up to 3.85$\times$ compared to the chunked prefill and 5.81$\times$ compared to the prefill-decoding disaggregation.

LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism

TL;DR

LoongServe introduces Elastic Sequence Parallelism (ESP) to dynamically adjust parallelism degree across iterations for long-context LLM serving, addressing variability in compute and memory demands. It implements zero-overhead elastic scale-down via proactive KV-cache migration during prefill and multi-master distributed decoding for scale-up, coupled with a four-step, real-time scheduling algorithm that optimizes dispatching, allocation, batching, and scaling plans. A unified distributed KV cache pool and carefully engineered routing and kernel optimizations enable efficient resource sharing and reduced memory fragmentation. Evaluations on long-context models and diverse workloads show up to 3.85× throughput gains over chunked prefill and 5.81× over prefill-decoding disaggregation, with robust multi-node scalability and strong analytical modeling for planning.

Abstract

The context window of large language models (LLMs) is rapidly increasing, leading to a huge variance in resource usage between different requests as well as between different phases of the same request. Restricted by static parallelism strategies, existing LLM serving systems cannot efficiently utilize the underlying resources to serve variable-length requests in different phases. To address this problem, we propose a new parallelism paradigm, elastic sequence parallelism (ESP), to elastically adapt to the variance between different requests and phases. Based on ESP, we design and build LoongServe, an LLM serving system that (1) improves computation efficiency by elastically adjusting the degree of parallelism in real-time, (2) improves communication efficiency by reducing key-value cache migration overhead and overlapping partial decoding communication with computation, and (3) improves GPU memory efficiency by reducing key-value cache fragmentation across instances. Our evaluation under diverse real-world datasets shows that LoongServe improves the maximum throughput by up to 3.85 compared to the chunked prefill and 5.81 compared to the prefill-decoding disaggregation.
Paper Structure (24 sections, 7 equations, 15 figures)

This paper contains 24 sections, 7 equations, 15 figures.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. The Process of LLM Inference
  4. Existing LLM Serving Systems
  5. Sequence Parallelism
  6. Motivation and Challenges
  7. LoongServe Overview
  8. LoongServe Elastic Instances
  9. Elastic Scale-down
  10. Elastic Scale-up
  11. LoongServe Global Manager
  12. Dispatching
  13. Elastic Instance Allocation
  14. Batching
  15. Elastic Scaling Plan Generation
  16. ...and 9 more sections

Figures (15)

  • Figure 1: Sequence parallelism in the prefill phase.
  • Figure 2: Scalability of requests with different lengths in the different phases.
  • Figure 3: Comparison between fixed sequence parallelism and tensor parallelism.
  • Figure 4: KV cache fragmentation of group based strategy.
  • Figure 5: LoongServe system overview.
  • ...and 10 more figures