Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hyperion: Hierarchical Scheduling for Parallel LLM Acceleration in Multi-tier Networks

Mulei Ma, Xinyi Xu, Minrui Xu, Zihan Chen, Yang Yang, Tony Q. S. Quek

TL;DR

Hyperion addresses the challenge of low-latency LLM inference on resource-constrained, multi-tier edge networks by decoupling the joint optimization of model partitioning and request scheduling into offline inter-tier partitioning (HypSplit-DP) and online intra-tier scheduling (HypSched-RT). The offline stage computes a globally balanced partition across tiers, while the online stage rapidly maps each incoming task to the best node within its tier, adapting to real-time load. Empirical results on Llama3-8B and Phi-3-medium show Hyperion achieving substantial latency reductions (up to about 52% vs GPipe and 31% vs HEFT) and improved GPU utilization, with clear scalability advantages for long-sequence generation and deeper, multi-tier networks. The work demonstrates that cross-tier, resource-aware orchestration is crucial for practical edge deployment of large language models.

Abstract

LLMs are increasingly executed in edge where limited GPU memory and heterogeneous computation jointly constrain deployment which motivates model partitioning and request scheduling. In this setting, minimizing latency requires addressing the tight coupling between model placement and request scheduling across heterogeneous nodes, as suboptimal decisions in one domain can negate benefits in the other. In this paper, we propose Hyperion, a hierarchical two-stage framework that jointly optimizes partitioning and scheduling for pipelined LLM inference. Hyperion minimizes latency by balancing resources across tiers without requiring model retraining or incurring significant runtime overhead. Leveraging the timescale difference between partitioning and request arrivals, Stage 1 performs offline, inter-tier partitioning via a Hyperion Split with Dynamic Programming (HypSplit-DP) procedure to produce balanced stage times under tier capacity and memory constraints; to adapt to time-varying load, Stage 2 performs online, intra-tier scheduling with a lightweight Hyperion Scheduling for Real-Time (HypSched-RT) that maps each request to the best available node using real-time estimates of queue length and effective capacity. Experiments with Phi-3-medium demonstrate that Hyperion reduces latency by up to 52.1% (vs. GPipe) and 31.2% (vs. HEFT). Furthermore, Hyperion exhibits superior scalability for long-sequence generation, maintaining 44.5% lower latency and higher GPU utilization.

Hyperion: Hierarchical Scheduling for Parallel LLM Acceleration in Multi-tier Networks

TL;DR

Hyperion addresses the challenge of low-latency LLM inference on resource-constrained, multi-tier edge networks by decoupling the joint optimization of model partitioning and request scheduling into offline inter-tier partitioning (HypSplit-DP) and online intra-tier scheduling (HypSched-RT). The offline stage computes a globally balanced partition across tiers, while the online stage rapidly maps each incoming task to the best node within its tier, adapting to real-time load. Empirical results on Llama3-8B and Phi-3-medium show Hyperion achieving substantial latency reductions (up to about 52% vs GPipe and 31% vs HEFT) and improved GPU utilization, with clear scalability advantages for long-sequence generation and deeper, multi-tier networks. The work demonstrates that cross-tier, resource-aware orchestration is crucial for practical edge deployment of large language models.

Abstract

LLMs are increasingly executed in edge where limited GPU memory and heterogeneous computation jointly constrain deployment which motivates model partitioning and request scheduling. In this setting, minimizing latency requires addressing the tight coupling between model placement and request scheduling across heterogeneous nodes, as suboptimal decisions in one domain can negate benefits in the other. In this paper, we propose Hyperion, a hierarchical two-stage framework that jointly optimizes partitioning and scheduling for pipelined LLM inference. Hyperion minimizes latency by balancing resources across tiers without requiring model retraining or incurring significant runtime overhead. Leveraging the timescale difference between partitioning and request arrivals, Stage 1 performs offline, inter-tier partitioning via a Hyperion Split with Dynamic Programming (HypSplit-DP) procedure to produce balanced stage times under tier capacity and memory constraints; to adapt to time-varying load, Stage 2 performs online, intra-tier scheduling with a lightweight Hyperion Scheduling for Real-Time (HypSched-RT) that maps each request to the best available node using real-time estimates of queue length and effective capacity. Experiments with Phi-3-medium demonstrate that Hyperion reduces latency by up to 52.1% (vs. GPipe) and 31.2% (vs. HEFT). Furthermore, Hyperion exhibits superior scalability for long-sequence generation, maintaining 44.5% lower latency and higher GPU utilization.

Paper Structure

This paper contains 21 sections, 21 equations, 16 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Distributed Accelerated Inference of LLMs in Edge Computing
  4. Resource Scheduling for LLMs in Edge Computing
  5. Parallel Computing of LLMs in Edge Computing
  6. Research on LLMs Scheduling in Multi-tier Computing Networks
  7. System Model and Problem Formulation
  8. Modeling of LLM Structure
  9. Multi-tier Heterogeneous Network Architecture
  10. Overall Optimization Problem
  11. Problem Decoupling and Hierarchical Solution
  12. HYPERION: A two-stage HIERARCHICAL PARTITIONING AND SCHEDULING FRAMEWORK
  13. Stage 1: Offline inter-tier LLM Partitioning via HypSplit-DP
  14. Stage 2: Online intra-tier Task Scheduling via HypSched-RT
  15. Experiment and Model Evaluation
  16. ...and 6 more sections

Figures (16)

  • Figure 1: Multi-tier Network Architecture Incorporating three Edge Tiers for Optimized LLM Inference and Resource Management
  • Figure 2: Hierarchical Architecture of a Llama 3, from Decoder Blocks to Detailed GQA and SwiGLU Component Views.
  • Figure 3: Framework for Problem Decomposition in Hyperion, Detailing how the Overall Optimization Problem ($\mathcal{P}_{0}$) is Decoupled into two Sub-problems, Inter-tier Partitioning ($\boldsymbol{\mathcal{P}_{1}}$) and Intra-tier Scheduling ($\boldsymbol{\mathcal{P}_{2}}$), and Solved Respectively.
  • Figure 4: System Architecture and Operational Workflow of the Hyperion Framework, Illustrating the end-to-end Process from Offline LLM Partitioning in the Cloud to Online, Real Time Task Execution Across the Multi-tier Network.
  • Figure 5: 1 Gbps
  • ...and 11 more figures

Theorems & Definitions (1)

  • Definition 1