Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimizing Resource Allocation for Geographically-Distributed Inference by Large Language Models

Tingyang Sun, Ting He, Bo Ji, Parimal Parag

TL;DR

The paper tackles the challenge of optimizing resource allocation for geographically distributed LLM inference by jointly placing transformer blocks and routing requests across multiple servers. It introduces experimentally validated performance models and proves the NP-hardness of the offline BPRR problem, then delivers a polynomial-complexity CG-BPRR algorithm with guaranteed average per-token inference time. The authors extend the approach online via a robust block-placement and waiting-aware routing framework, achieving bounded performance under bounded load. Extensive experiments on PETALS and a CPU-only simulator demonstrate substantial reductions in inference time, especially for the first token, and provide practical insights for deployment with limited GPU access. The work offers a practical, scalable toolkit for accelerating distributed LLM inference in real-world, geographically dispersed settings.

Abstract

Large language models have demonstrated extraordinary performance in many AI tasks but are expensive to use, even after training, due to their requirement of high-end GPUs. Recently, a distributed system called PETALS was developed to lower the barrier for deploying LLMs by splitting the model blocks across multiple servers with low-end GPUs distributed over the Internet, which was much faster than swapping the model parameters between the GPU memory and other cheaper but slower local storage media. However, the performance of such a distributed system critically depends on the resource allocation, and how to do so optimally remains unknown. In this work, we present the first systematic study of the resource allocation problem in distributed LLM inference, with focus on two important decisions: block placement and request routing. Our main results include: experimentally validated performance models that can predict the inference performance under given block placement and request routing decisions, a formulation of the offline optimization of block placement and request routing as a mixed integer linear programming problem together with the NP-hardness proof and a polynomial-complexity algorithm with guaranteed performance, and an adaptation of the offline algorithm for the online setting with the same performance guarantee under bounded load. Through both experiments and experimentally-validated simulations, we have verified that the proposed solution can substantially reduce the inference time compared to the state-of-the-art solution in diverse settings with geographically-distributed servers. As a byproduct, we have also developed a light-weighted CPU-only simulator capable of predicting the performance of distributed LLM inference on GPU servers, which can evaluate large deployments and facilitate future research for researchers with limited GPU access.

Optimizing Resource Allocation for Geographically-Distributed Inference by Large Language Models

TL;DR

The paper tackles the challenge of optimizing resource allocation for geographically distributed LLM inference by jointly placing transformer blocks and routing requests across multiple servers. It introduces experimentally validated performance models and proves the NP-hardness of the offline BPRR problem, then delivers a polynomial-complexity CG-BPRR algorithm with guaranteed average per-token inference time. The authors extend the approach online via a robust block-placement and waiting-aware routing framework, achieving bounded performance under bounded load. Extensive experiments on PETALS and a CPU-only simulator demonstrate substantial reductions in inference time, especially for the first token, and provide practical insights for deployment with limited GPU access. The work offers a practical, scalable toolkit for accelerating distributed LLM inference in real-world, geographically dispersed settings.

Abstract

Large language models have demonstrated extraordinary performance in many AI tasks but are expensive to use, even after training, due to their requirement of high-end GPUs. Recently, a distributed system called PETALS was developed to lower the barrier for deploying LLMs by splitting the model blocks across multiple servers with low-end GPUs distributed over the Internet, which was much faster than swapping the model parameters between the GPU memory and other cheaper but slower local storage media. However, the performance of such a distributed system critically depends on the resource allocation, and how to do so optimally remains unknown. In this work, we present the first systematic study of the resource allocation problem in distributed LLM inference, with focus on two important decisions: block placement and request routing. Our main results include: experimentally validated performance models that can predict the inference performance under given block placement and request routing decisions, a formulation of the offline optimization of block placement and request routing as a mixed integer linear programming problem together with the NP-hardness proof and a polynomial-complexity algorithm with guaranteed performance, and an adaptation of the offline algorithm for the online setting with the same performance guarantee under bounded load. Through both experiments and experimentally-validated simulations, we have verified that the proposed solution can substantially reduce the inference time compared to the state-of-the-art solution in diverse settings with geographically-distributed servers. As a byproduct, we have also developed a light-weighted CPU-only simulator capable of predicting the performance of distributed LLM inference on GPU servers, which can evaluate large deployments and facilitate future research for researchers with limited GPU access.
Paper Structure (35 sections, 8 theorems, 34 equations, 20 figures, 10 tables, 2 algorithms)

This paper contains 35 sections, 8 theorems, 34 equations, 20 figures, 10 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Summary of Contributions
  4. Problem Formulation
  5. Target Application Scenario
  6. System Model
  7. Resource Allocation Problem
  8. Joint Block Placement and Request Routing (BPRR)
  9. Preliminaries
  10. Offline Setting
  11. Vanilla Formulation
  12. MILP Formulation
  13. Algorithm Design
  14. Performance Analysis
  15. Online Setting
  16. ...and 20 more sections

Key Result

Lemma 3.1

Assume that each node $j\in V$ stores all the blocks in $\{a_j,\ldots,a_j+m_j-1\}$, where each S-client $c\in V^S_c$ stores a dummy block $0$ (i.e., $a_c:=0$, $m_c:=1$), and each D-client $c'\in V^D_c$ stores another dummy block $L+1$ (i.e., $a_{c'}:=L+1$, $m_{c'}:=1$). Then a $c$-to-$c'$ path $p$ i

Figures (20)

  • Figure 1: System architecture for pipeline-parallel LLM inference using client-centric communication.
  • Figure 2: Inference time vs. #processed blocks on A100 for: (a) first token; (b) each of remaining tokens ($l_{\text{max}}^I=20,\ l_{\text{max}}=128$).
  • Figure 3: Memory consumption vs. #blocks on A100 for: (a) total memory usage; (b) attention caches ($l_{\text{max}}^I=20,\: l_{\text{max}} = 128$).
  • Figure 4: Logical topology $G$ for request routing and route feasibility condition.
  • Figure 5: Example for suboptimality of CG-BPRR (Alg. \ref{['Alg:CG-BPRR']}).
  • ...and 15 more figures

Theorems & Definitions (16)

  • Lemma 3.1
  • Theorem 3.2
  • Lemma 3.3
  • Lemma 3.4
  • Theorem 3.5
  • Corollary 3.6
  • Corollary 3.7
  • proof : Proof of Lemma \ref{['lem:chain feasibility constraint']}
  • proof : Proof of Theorem \ref{['thm:NP-hardness of BPRR']}
  • proof : Proof of Lemma \ref{['lem:minimize inference time under relaxed routing']}
  • ...and 6 more