Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Routing of Inference Requests across LLM Instances in Cloud-Edge Computing

Shibo Yu, Mohammad Goudarzi, Adel Nadjaran Toosi

TL;DR

This work targets scalable and cost-efficient LLM inference in cloud–edge environments by formulating inference routing as a multi-objective optimization problem over response quality, latency, and cost. It introduces an NSGA-II based routing algorithm that assigns heterogeneous requests to heterogeneous LLMs across edge and cloud nodes, yielding a Pareto front of routing policies. Empirical evaluation on mixed benchmarks (SQuAD, MBPP, HellaSwag, GSM8K) shows the approach achieves 95.2% of Cloud-Only quality with only a modest latency increase while reducing inference cost by 34.9%, outperforming several baseline routing strategies. The work demonstrates a practical framework for adaptive, distributed LLM inference in resource-constrained, dynamic cloud–edge settings, with potential extensions to real-time monitoring and multimodal inputs.

Abstract

The rising demand for Large Language Model (LLM) inference services has intensified pressure on computational resources, resulting in latency and cost challenges. This paper introduces a novel routing algorithm based on the Non-dominated Sorting Genetic Algorithm II (NSGA-II) to distribute inference requests across heterogeneous LLM instances in a cloud-edge computing environment. Formulated as a multi-objective optimization problem, the algorithm balances response quality, response time, and inference cost, adapting to request heterogeneity (e.g., varying complexity and prompt lengths) and node diversity (e.g., edge vs. cloud resources). This adaptive routing algorithm optimizes performance under dynamic workloads. We benchmark the approach using a testbed with datasets including Stanford Question Answering Dataset (SQuAD), Mostly Basic Python Problems (MBPP), Hella Situations With Adversarial Generations (HellaSwag), and Grade School Math 8K (GSM8K). Experimental results show our solution, compared to the baselines, preserves 95.2% of Cloud-Only response quality with slight latency increase, while reducing inference cost by 34.9%. These findings validate the algorithm's effectiveness for scalable LLM deployments.

Efficient Routing of Inference Requests across LLM Instances in Cloud-Edge Computing

TL;DR

This work targets scalable and cost-efficient LLM inference in cloud–edge environments by formulating inference routing as a multi-objective optimization problem over response quality, latency, and cost. It introduces an NSGA-II based routing algorithm that assigns heterogeneous requests to heterogeneous LLMs across edge and cloud nodes, yielding a Pareto front of routing policies. Empirical evaluation on mixed benchmarks (SQuAD, MBPP, HellaSwag, GSM8K) shows the approach achieves 95.2% of Cloud-Only quality with only a modest latency increase while reducing inference cost by 34.9%, outperforming several baseline routing strategies. The work demonstrates a practical framework for adaptive, distributed LLM inference in resource-constrained, dynamic cloud–edge settings, with potential extensions to real-time monitoring and multimodal inputs.

Abstract

The rising demand for Large Language Model (LLM) inference services has intensified pressure on computational resources, resulting in latency and cost challenges. This paper introduces a novel routing algorithm based on the Non-dominated Sorting Genetic Algorithm II (NSGA-II) to distribute inference requests across heterogeneous LLM instances in a cloud-edge computing environment. Formulated as a multi-objective optimization problem, the algorithm balances response quality, response time, and inference cost, adapting to request heterogeneity (e.g., varying complexity and prompt lengths) and node diversity (e.g., edge vs. cloud resources). This adaptive routing algorithm optimizes performance under dynamic workloads. We benchmark the approach using a testbed with datasets including Stanford Question Answering Dataset (SQuAD), Mostly Basic Python Problems (MBPP), Hella Situations With Adversarial Generations (HellaSwag), and Grade School Math 8K (GSM8K). Experimental results show our solution, compared to the baselines, preserves 95.2% of Cloud-Only response quality with slight latency increase, while reducing inference cost by 34.9%. These findings validate the algorithm's effectiveness for scalable LLM deployments.

Paper Structure

This paper contains 29 sections, 14 equations, 4 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Cloud and Edge Collaboration
  4. LLMs and Transformer Architecture
  5. Metaheuristic in Multi-objective Optimization
  6. Related Work
  7. Problem Formulation
  8. Response Quality Metric (RQ)
  9. Inference Cost Metric (C)
  10. Response Time Metric (RT)
  11. Discussion
  12. Proposed Solution
  13. System Architecture
  14. NSGA-II-Based Routing Algorithm
  15. Population Initialization
  16. ...and 14 more sections

Figures (4)

  • Figure 1: System architecture comprising cloud and edge nodes with request routing via NSGA-II.
  • Figure 2: Average quality per dataset for different routing strategies. Our solution achieves competitive quality across MBPP, GSM8K, SQuAD, and HellaSwag.
  • Figure 3: Three-dimensional performance comparison of routing strategies, illustrating trade-offs among model performance (quality), response time, and cost.
  • Figure 4: Performance of our proposed router at different concurrency levels (1, 4, 8, 10) in terms of average quality, response time, and cost.