PerLLM: Personalized Inference Scheduling with Edge-Cloud Collaboration for Diverse LLM Services

TL;DR

PerLLM addresses real-time, energy-efficient inference for diverse LLM services under bandwidth constraints by scheduling tasks across edge and cloud resources. It casts the problem as a combinatorial multi-armed bandit with a constraint-satisfaction mechanism and solves it with a CS-UCB algorithm that optimizes the objective $\min \frac{1}{T} \sum_{t=0}^T (\omega_{\text{tran}} E_{\text{tran}}^t + \omega_{\text{infer}} E_{\text{infer}}^t + \omega_{\text{idle}} E_{\text{idle}}^t)$ subject to constraints. The theory provides a regret bound $Reg(T) \leq \sqrt{2 M N \log (L)} + \theta P(t)$ and linear-time per-update complexity, while experiments show throughput gains of up to 2.2× and energy savings exceeding 50% across models and bandwidth scenarios. This work demonstrates practical benefits for scalable, personalized LLM inference in edge-cloud ecosystems.

Abstract

With the rapid growth in the number of large language model (LLM) users, it is difficult for bandwidth-constrained cloud servers to simultaneously process massive LLM services in real-time. Recently, edge-cloud infrastructures have been used to improve the processing efficiency of large-scale LLM services. However, the diversity of task requirements and the dynamics of resources pose great challenges to inference scheduling, leading to the wastage of many resources. In this paper, we present PerLLM, a personalized inference scheduling framework with edge-cloud collaboration designed for diverse LLM services. For the complexity of multiple constraints and the decision-making process of edge-cloud collaboration, we integrate the upper confidence bound algorithm based on the constraint satisfaction mechanism in PerLLM. For diverse LLM services, PerLLM can optimize service scheduling and resource allocation solutions within the edge-cloud infrastructure to meet processing time requirements while minimizing energy costs. Experimental results from different model deployments show that PerLLM can effectively meet the processing time requirements of personalized services. Compared to other methods, PerLLM achieves 2.2x, 2.1x, and 1.6x throughput and reduces the energy cost by more than 50%.

Figures (6)

• Figure 1: The illustration of edge-cloud collaborative inference architecture for LLM services.
• Figure 2: The test results of average processing time and energy costs on cloud and edge.
• Figure 3: The overall design and workflow of the proposed PerLLM framework.
• Figure 4: The average processing time comparison of different methods under different models.
• Figure 5: The throughput comparison of different methods under different models.