Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference

Hyungjun Oh, Kihong Kim, Jaemin Kim, Sungkyun Kim, Junyeol Lee, Du-seong Chang, Jiwon Seo

TL;DR

ExeGPT tackles constraint-aware LLM inference by decoupling encoding and decoding and optimizing GPU resource allocation via two strategies, RRA and WAA, guided by a monotonic-branch-and-bound scheduler. It leverages sequence-length distributions through XProfiler/XSimulator to predict throughput and latency, enabling efficient scheduling under latency constraints. Across six LLMs and five NLP tasks, ExeGPT achieves up to 15.2× throughput improvements and up to 6× latency reductions versus FasterTransformer, averaging 2.9× throughput gains. The method adapts to distribution changes with moderate reconfiguration costs, making it practical for real-world serving with diverse SLAs and workload patterns.

Abstract

This paper presents ExeGPT, a distributed system designed for constraint-aware LLM inference. ExeGPT finds and runs with an optimal execution schedule to maximize inference throughput while satisfying a given latency constraint. By leveraging the distribution of input and output sequences, it effectively allocates resources and determines optimal execution configurations, including batch sizes and partial tensor parallelism. We also introduce two scheduling strategies based on Round-Robin Allocation and Workload-Aware Allocation policies, suitable for different NLP workloads. We evaluate ExeGPT on six LLM instances of T5, OPT, and GPT-3 and five NLP tasks, each with four distinct latency constraints. Compared to FasterTransformer, ExeGPT achieves up to 15.2x improvements in throughput and 6x improvements in latency. Overall, ExeGPT achieves an average throughput gain of 2.9x across twenty evaluation scenarios. Moreover, when adapting to changing sequence distributions, the cost of adjusting the schedule in ExeGPT is reasonably modest. ExeGPT proves to be an effective solution for optimizing and executing LLM inference for diverse NLP workload and serving conditions.

ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference

TL;DR

ExeGPT tackles constraint-aware LLM inference by decoupling encoding and decoding and optimizing GPU resource allocation via two strategies, RRA and WAA, guided by a monotonic-branch-and-bound scheduler. It leverages sequence-length distributions through XProfiler/XSimulator to predict throughput and latency, enabling efficient scheduling under latency constraints. Across six LLMs and five NLP tasks, ExeGPT achieves up to 15.2× throughput improvements and up to 6× latency reductions versus FasterTransformer, averaging 2.9× throughput gains. The method adapts to distribution changes with moderate reconfiguration costs, making it practical for real-world serving with diverse SLAs and workload patterns.

Abstract

This paper presents ExeGPT, a distributed system designed for constraint-aware LLM inference. ExeGPT finds and runs with an optimal execution schedule to maximize inference throughput while satisfying a given latency constraint. By leveraging the distribution of input and output sequences, it effectively allocates resources and determines optimal execution configurations, including batch sizes and partial tensor parallelism. We also introduce two scheduling strategies based on Round-Robin Allocation and Workload-Aware Allocation policies, suitable for different NLP workloads. We evaluate ExeGPT on six LLM instances of T5, OPT, and GPT-3 and five NLP tasks, each with four distinct latency constraints. Compared to FasterTransformer, ExeGPT achieves up to 15.2x improvements in throughput and 6x improvements in latency. Overall, ExeGPT achieves an average throughput gain of 2.9x across twenty evaluation scenarios. Moreover, when adapting to changing sequence distributions, the cost of adjusting the schedule in ExeGPT is reasonably modest. ExeGPT proves to be an effective solution for optimizing and executing LLM inference for diverse NLP workload and serving conditions.
Paper Structure (21 sections, 2 equations, 11 figures, 7 tables, 1 algorithm)

This paper contains 21 sections, 2 equations, 11 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Overview of ExeGPT
  4. Scheduling Strategies and Trade-offs
  5. Decoupled Scheduling of Encoders and Decoders
  6. Controlling Latency and Throughput Trade-off
  7. Constraint-Aware Scheduling Algorithm
  8. Scheduling Algorithm for Monotonic Optimization
  9. Dynamic Scheduling for Consistent Workload
  10. Simulation with Sequence Distribution
  11. Evaluation
  12. Evaluation Methodology
  13. Performance Comparison of Existing Systems
  14. Performance Evaluation of Small to Mid-sized LLMs
  15. Performance Evaluation of Large LLMs
  16. ...and 6 more sections

Figures (11)

  • Figure 1: Illustration of a Transformer decoder (left) and inference timelines of three LLM inference systems (right).
  • Figure 2: High-level architecture of ExeGPT
  • Figure 3: Comparing RRA and WAA Scheduling strategies
  • Figure 4: Execution timelines of RRA/WAA Scheduling with GPU$_1$--GPU$_4$. The blue/red boxes are encoding/decoding iterations, respectively. The numbers are mini-batch IDs.
  • Figure 5: Illustration of the branch-and-bound scheduling. The filled circles are configuration points within the latency bound and empty circles are those that are not. $T^*$ is the current optimal point.
  • ...and 6 more figures