High-Throughput LLM inference on Heterogeneous Clusters

TL;DR

This work tackles high-throughput LLM inference on heterogeneous clusters by separating deployment optimization from runtime scheduling. It introduces a lightweight profiling-based method to optimally allocate tensor parallelism and instances per machine, coupled with a capacity-aware runtime scheduler that accounts for both compute and memory (KV cache) constraints. Empirical results on two-machine and multi-machine setups show substantial throughput gains, including up to 122.5% and 33.6% improvements, respectively. The approach enhances resource utilization and reduces load imbalance, offering practical benefits for cost-efficient LLM serving in heterogeneous environments.

Abstract

Nowadays, many companies possess various types of AI accelerators, forming heterogeneous clusters. Efficiently leveraging these clusters for high-throughput large language model (LLM) inference services can significantly reduce costs and expedite task processing. However, LLM inference on heterogeneous clusters presents two main challenges. Firstly, different deployment configurations can result in vastly different performance. The number of possible configurations is large, and evaluating the effectiveness of a specific setup is complex. Thus, finding an optimal configuration is not an easy task. Secondly, LLM inference instances within a heterogeneous cluster possess varying processing capacities, leading to different processing speeds for handling inference requests. Evaluating these capacities and designing a request scheduling algorithm that fully maximizes the potential of each instance is challenging. In this paper, we propose a high-throughput inference service system on heterogeneous clusters. First, the deployment configuration is optimized by modeling the resource amount and expected throughput and using the exhaustive search method. Second, a novel mechanism is proposed to schedule requests among instances, which fully considers the different processing capabilities of various instances. Extensive experiments show that the proposed scheduler improves throughput by 122.5% and 33.6% on two heterogeneous clusters, respectively.

Figures (6)

• Figure 1: An illustration of the system architecture. First, we perform deployment configuration optimization to identify the optimal deployment setup. Next, a runtime request scheduler is employed to distribute requests among instances.
• Figure 2: An illustration of the prefill phase and decoding phase.
• Figure 3: An illustration of the scheduler framework, in which (1) represents requests information, (2) and (3) means instances' fitted parameter, (4) means history mappings, and (5) means calculated workloads.
• Figure 4: System throughput and instance completion time distribution under different deployment configurations. The first subgraph shows the throughput under two different random seeds and two estimated results. The ranking of deployment configurations in terms of throughput remains consistent between the actual results and the estimates. The second subgraph illustrates the completion times of each instance in the system under the first random seed. With the guarantee of our designed request scheduling mechanism, there is no workload imbalance in the system.
• Figure 5: The throughput and instance completion time distribution for a system with two instances, one with t=4 and the other with t=1, demonstrate that the Round Robin scheduling strategy leads to extreme workload imbalance between instances, which becomes a bottleneck for system throughput. In contrast, our scheduler significantly reduces this imbalance.