Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UELLM: A Unified and Efficient Approach for LLM Inference Serving

Yiyuan He, Minxian Xu, Jingfeng Wu, Wanyi Zheng, Kejiang Ye, Chengzhong Xu

TL;DR

The paper tackles efficient LLM inference serving in MLaaS clouds under heavy query loads and strict latency constraints. It introduces UELLM, a unified framework consisting of a Resource Profiler, a SLO-aware Batch Scheduler (SLO-ODBS), and an LLM Deployer (HELR) that leverages dynamic programming for device mapping, together minimizing memory overhead and inference latency while reducing SLO violations. The approach yields substantial gains over state-of-the-art baselines, achieving latency reductions of $72.3\%$ to $90.3\%$, GPU utilization improvements of $1.2\times$ to $4.1\times$, and throughput increases of $1.92\times$ to $4.98\times$, with all SLOs met in experiments. The work demonstrates practical end-to-end optimization for Transformer-based LLM inference in cloud environments and provides an open-source implementation for broader adoption.

Abstract

In the context of Machine Learning as a Service (MLaaS) clouds, the extensive use of Large Language Models (LLMs) often requires efficient management of significant query loads. When providing real-time inference services, several challenges arise. Firstly, increasing the number of GPUs may lead to a decrease in inference speed due to heightened communication overhead, while an inadequate number of GPUs can lead to out-of-memory errors. Secondly, different deployment strategies need to be evaluated to guarantee optimal utilization and minimal inference latency. Lastly, inefficient orchestration of inference queries can easily lead to significant Service Level Objective (SLO) violations. Lastly, inefficient orchestration of inference queries can easily lead to significant Service Level Objective (SLO) violations. To address these challenges, we propose a Unified and Efficient approach for Large Language Model inference serving (UELLM), which consists of three main components: 1) resource profiler, 2) batch scheduler, and 3) LLM deployer. UELLM minimizes resource overhead, reduces inference latency, and lowers SLO violation rates. Compared with state-of-the-art (SOTA) techniques, UELLM reduces the inference latency by 72.3% to 90.3%, enhances GPU utilization by 1.2X to 4.1X, and increases throughput by 1.92X to 4.98X, it can also serve without violating the inference latency SLO.

UELLM: A Unified and Efficient Approach for LLM Inference Serving

TL;DR

The paper tackles efficient LLM inference serving in MLaaS clouds under heavy query loads and strict latency constraints. It introduces UELLM, a unified framework consisting of a Resource Profiler, a SLO-aware Batch Scheduler (SLO-ODBS), and an LLM Deployer (HELR) that leverages dynamic programming for device mapping, together minimizing memory overhead and inference latency while reducing SLO violations. The approach yields substantial gains over state-of-the-art baselines, achieving latency reductions of to , GPU utilization improvements of to , and throughput increases of to , with all SLOs met in experiments. The work demonstrates practical end-to-end optimization for Transformer-based LLM inference in cloud environments and provides an open-source implementation for broader adoption.

Abstract

In the context of Machine Learning as a Service (MLaaS) clouds, the extensive use of Large Language Models (LLMs) often requires efficient management of significant query loads. When providing real-time inference services, several challenges arise. Firstly, increasing the number of GPUs may lead to a decrease in inference speed due to heightened communication overhead, while an inadequate number of GPUs can lead to out-of-memory errors. Secondly, different deployment strategies need to be evaluated to guarantee optimal utilization and minimal inference latency. Lastly, inefficient orchestration of inference queries can easily lead to significant Service Level Objective (SLO) violations. Lastly, inefficient orchestration of inference queries can easily lead to significant Service Level Objective (SLO) violations. To address these challenges, we propose a Unified and Efficient approach for Large Language Model inference serving (UELLM), which consists of three main components: 1) resource profiler, 2) batch scheduler, and 3) LLM deployer. UELLM minimizes resource overhead, reduces inference latency, and lowers SLO violation rates. Compared with state-of-the-art (SOTA) techniques, UELLM reduces the inference latency by 72.3% to 90.3%, enhances GPU utilization by 1.2X to 4.1X, and increases throughput by 1.92X to 4.98X, it can also serve without violating the inference latency SLO.
Paper Structure (18 sections, 6 equations, 5 figures, 2 tables, 2 algorithms)

This paper contains 18 sections, 6 equations, 5 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Background: Generative LLM Consumes Large Amount of Memory
  4. Motivation: Deployment Configuration versus LLM Inference Performance
  5. Motivation: Batching versus LLM Inference Performance
  6. Related Work
  7. Model Deployment
  8. Model Inference
  9. System Design
  10. Resource Profiler
  11. Batch Scheduler
  12. LLM Deployer
  13. Implementation and Evaluations
  14. Evaluation Setup
  15. Baselines and Metrics
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Normalized latency, memory usage, and GPU utilization under different configurations of GPU numbers and batch sizes. Each metric is normalized to its minimum value.
  • Figure 2: The main framework and functional details of UELLM.
  • Figure 3: Comparison between UELLM and the default batching algorithm. The comparison illustrates the utilization of the KV Cache in both the batching and inference stages for three queries. UELLM focuses on optimizing token usage, leading to a notable decrease in the overall number of tokens processed during inference in comparison to the default method.
  • Figure 4: Performance comparison of different batching algorithms and deployment algorithms. (a) Latency, (b) SLO violation under various batching algorithms, while (c) Throughput, and (d) GPU utilization under different deployment algorithms.
  • Figure 5: Comparison results of various metrics between $S^3$, Morphling, UD, UB, and UA. Metrics include (a) GPU utilization, (b) SLO non-violation, (c) inference latency, and (d) throughput.