Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization

Juntao Zhao, Borui Wan, Yanghua Peng, Haibin Lin, Chuan Wu

TL;DR

LLM-PQ is proposed, a system that advocates adaptive model quantization and phase-aware partition to improve LLM serving efficiency on heterogeneous GPU clusters and carefully decides on mixed-precision model quantization together with phase-aware model partition and micro-batch sizing in distributed LLM serving with an efficient algorithm.

Abstract

Recent breakthroughs in Large-scale language models (LLMs) have demonstrated impressive performance on various tasks. The immense sizes of LLMs have led to very high resource demand and cost for running the models. Though the models are largely served using uniform high-caliber GPUs nowadays, utilizing a heterogeneous cluster with a mix of available high- and low-capacity GPUs can potentially substantially reduce the serving cost. There is a lack of designs to support efficient LLM serving using a heterogeneous cluster, while the current solutions focus on model partition and uniform compression among homogeneous devices. This paper proposes LLM-PQ, a system that advocates adaptive model quantization and phase-aware partition to improve LLM serving efficiency on heterogeneous GPU clusters. We carefully decide on mixed-precision model quantization together with phase-aware model partition and micro-batch sizing in distributed LLM serving with an efficient algorithm, to greatly enhance inference throughput while fulfilling user-specified model quality targets. Extensive experiments on production inference workloads in 11 different clusters demonstrate that LLM-PQ achieves up to 2.88x (2.26x on average) throughput improvement in inference, showing great advantages over state-of-the-art works.

LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization

TL;DR

LLM-PQ is proposed, a system that advocates adaptive model quantization and phase-aware partition to improve LLM serving efficiency on heterogeneous GPU clusters and carefully decides on mixed-precision model quantization together with phase-aware model partition and micro-batch sizing in distributed LLM serving with an efficient algorithm.

Abstract

Recent breakthroughs in Large-scale language models (LLMs) have demonstrated impressive performance on various tasks. The immense sizes of LLMs have led to very high resource demand and cost for running the models. Though the models are largely served using uniform high-caliber GPUs nowadays, utilizing a heterogeneous cluster with a mix of available high- and low-capacity GPUs can potentially substantially reduce the serving cost. There is a lack of designs to support efficient LLM serving using a heterogeneous cluster, while the current solutions focus on model partition and uniform compression among homogeneous devices. This paper proposes LLM-PQ, a system that advocates adaptive model quantization and phase-aware partition to improve LLM serving efficiency on heterogeneous GPU clusters. We carefully decide on mixed-precision model quantization together with phase-aware model partition and micro-batch sizing in distributed LLM serving with an efficient algorithm, to greatly enhance inference throughput while fulfilling user-specified model quality targets. Extensive experiments on production inference workloads in 11 different clusters demonstrate that LLM-PQ achieves up to 2.88x (2.26x on average) throughput improvement in inference, showing great advantages over state-of-the-art works.
Paper Structure (30 sections, 2 theorems, 5 equations, 9 figures, 10 tables, 2 algorithms)

This paper contains 30 sections, 2 theorems, 5 equations, 9 figures, 10 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. BackGround and Motivation
  3. Generative Inference of LLM
  4. Heterogenous Model Parallelization
  5. Online and Offline Serving Task
  6. Quantization in LLM
  7. Challenges
  8. LLM-PQ Overview
  9. Assigner Design
  10. Cost Model
  11. Indicator of Model Perturbation by Quantization
  12. Optimizer
  13. Implementation
  14. Evaluation
  15. Experimental Setup
  16. ...and 15 more sections

Key Result

Theorem 1

The variance of a linear operator's output after weight-only quantization using stochastic or deterministic rounding is: where $D_\mathbf{W}$ is the dimension of model weights $\mathbf{W}$ and $S_\mathbf{W}$ is the scaling factor.

Figures (9)

  • Figure 1: GPU proportions and utilization rates in a real-world production AI cluster.
  • Figure 2: Two phases in LLM generative serving: (Top) Prefill phase takes the prompt sequence to generate the initial key-value pairs. (Bottom) Decode phase takes previously generated token & stored KV pairs to generate the next token.
  • Figure 3: Phase time decomposition with different precisions. $\times$ indicates time on P100 compared to V100.
  • Figure 4: BLOOM-3b (a) and OPT-1.3b (b) perplexity (PPL) & accuracy under different quantization schemes. Smaller PPL means the model is more confident in its prediction.
  • Figure 5: Execution time of prefill and decode phases under different precisions and batch sizes.
  • ...and 4 more figures

Theorems & Definitions (2)

  • Theorem 1
  • proposition 1: Variance Indicator