Hybrid Offline-online Scheduling Method for Large Language Model Inference Optimization

TL;DR

The paper addresses the challenge of optimizing LLM inference serving to maximize hardware utilization and throughput. It introduces a hybrid offline-online scheduling framework built on a mixed-integer programming model, with an offline Minimizing Makespan Bin Packing component and online sorting/preemption plus a Lagrangian-based iteration scheduler to decide between prefill and decode tasks. The approach yields tangible gains: hardware utilization rises from 80.2% to 89.1% and total inference time drops from 201.00s to 190.58s on the LLaMA-65B/GSM8K setup, with a 100-case study showing an average 8.0% improvement in utilization; a theoretical lower bound of $T^{LB}$ around 180s is used to quantify optimality, reducing the primal-dual gap by over 50%. These results demonstrate the practical impact of formally modeling LLM inference scheduling and using a two-stage solution to meet millisecond-scale online decision requirements, informing scalable deployment of large models.

Abstract

With the development of large language models (LLMs), it has become increasingly important to optimize hardware usage and improve throughput. In this paper, we study the inference optimization of the serving system that deploys LLMs. To optimize system throughput and maximize hardware utilization, we formulate the inference optimization problem as a mixed-integer programming (MIP) model and propose a hybrid offline-online method as solution. The offline method improves large-scale inference systems by introducing a Minimizing Makespan Bin Packing Problem. We further provide a theoretical lower bound computation method. Then, we propose an online sorting and preemptive scheduling method to better utilize hardware. In the online iteration scheduling process, a Lagrangian method is applied to evaluate the cost efficiency of inserting prefill stages versus decode stages at each iteration and dynamically determine when to preempt decoding tasks and insert prefill tasks. Experiments using real-world data from the LLaMA-65B model and the GSM8K dataset demonstrate that system utilization improves from 80.2% to 89.1%, and the total inference time decreases from 201.00 to 190.58 seconds. A 100-cases study shows that our method consistently outperforms the baseline method and improves the utilization rate by 8.0% on average. Finally, we discuss potential future extensions, including stochastic modeling, reinforcement learning-based schedulers, and dynamic decision-making strategies for system throughput and hardware utilization.

Figures (15)

• Figure 1: Illustration of LLM inference service system
• Figure 2: Illustration of LLM inference
• Figure 3: Illustration of PD Competition
• Figure 4: Online scheduling
• Figure 5: Illustration of Solution Method