Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EPARA: Parallelizing Categorized AI Inference in Edge Clouds

Yubo Wang, Yubo Cui, Tuo Shi, Danyang Li, Wenxin Li, Lide Suo, Tao Wang, Xin Xie

TL;DR

EPARA tackles the challenge of scalable edge-cloud AI inference by introducing task-categorized parallelism, a distributed request handler, and a state-aware submodular service placement mechanism. It enables both request-level (latency vs. frequency sensitivity) and service-level (single vs. multi-GPU tasks) resource allocation, integrated through lightweight information synchronization. Real-world testbeds and large-scale simulations show up to $2.1\times$ goodput gains and robust performance across diverse AI workloads, including LLMs and segmentation, with strong resilience to synchronization and hardware faults. The work demonstrates that end-to-end edge AI inference can be significantly improved by co-designing allocation, request handling, and placement, enabling practical deployment on heterogeneous edge hardware.

Abstract

With the increasing adoption of AI applications such as large language models and computer vision AI, the computational demands on AI inference systems are continuously rising, making the enhancement of task processing capacity using existing hardware a primary objective in edge clouds. We propose EPARA, an end-to-end AI parallel inference framework in edge, aimed at enhancing the edge AI serving capability. Our key idea is to categorize tasks based on their sensitivity to latency/frequency and requirement for GPU resources, thereby achieving both request-level and service-level task-resource allocation. EPARA consists of three core components: 1) a task-categorized parallelism allocator that decides the parallel mode of each task, 2) a distributed request handler that performs the calculation for the specific request, and 3) a state-aware scheduler that periodically updates service placement in edge clouds. We implement a EPARA prototype and conduct a case study on the EPARA operation for LLMs and segmentation tasks. Evaluation through testbed experiments involving edge servers, embedded devices, and microcomputers shows that EPARA achieves up to 2.1$\times$ higher goodput in production workloads compared to prior frameworks, while adapting to various edge AI inference tasks.

EPARA: Parallelizing Categorized AI Inference in Edge Clouds

TL;DR

EPARA tackles the challenge of scalable edge-cloud AI inference by introducing task-categorized parallelism, a distributed request handler, and a state-aware submodular service placement mechanism. It enables both request-level (latency vs. frequency sensitivity) and service-level (single vs. multi-GPU tasks) resource allocation, integrated through lightweight information synchronization. Real-world testbeds and large-scale simulations show up to goodput gains and robust performance across diverse AI workloads, including LLMs and segmentation, with strong resilience to synchronization and hardware faults. The work demonstrates that end-to-end edge AI inference can be significantly improved by co-designing allocation, request handling, and placement, enabling practical deployment on heterogeneous edge hardware.

Abstract

With the increasing adoption of AI applications such as large language models and computer vision AI, the computational demands on AI inference systems are continuously rising, making the enhancement of task processing capacity using existing hardware a primary objective in edge clouds. We propose EPARA, an end-to-end AI parallel inference framework in edge, aimed at enhancing the edge AI serving capability. Our key idea is to categorize tasks based on their sensitivity to latency/frequency and requirement for GPU resources, thereby achieving both request-level and service-level task-resource allocation. EPARA consists of three core components: 1) a task-categorized parallelism allocator that decides the parallel mode of each task, 2) a distributed request handler that performs the calculation for the specific request, and 3) a state-aware scheduler that periodically updates service placement in edge clouds. We implement a EPARA prototype and conduct a case study on the EPARA operation for LLMs and segmentation tasks. Evaluation through testbed experiments involving edge servers, embedded devices, and microcomputers shows that EPARA achieves up to 2.1 higher goodput in production workloads compared to prior frameworks, while adapting to various edge AI inference tasks.

Paper Structure

This paper contains 27 sections, 2 theorems, 5 equations, 20 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Motivation
  3. Background and Limitations
  4. What the AI Inference Tasks Really Need in Edge Clouds?
  5. The Design Space and Challenges
  6. Design
  7. Task-Categorized Parallelism Allocator
  8. Distributed Request Handler
  9. State-aware Submodular Service Placement
  10. Synchronization, Granularity and Flexibility
  11. Implementation
  12. Adaptive Deployment
  13. Management of Edge Clouds
  14. Case Study: LLMs from Chats to Robots
  15. Evaluation
  16. ...and 12 more sections

Key Result

theorem 1

E p a r a state-aware placement $\varphi(X)$ is a submodular function.

Figures (20)

  • Figure 1: Request-level and service-level allocation.
  • Figure 2: Examples of service-level AI parallelism.
  • Figure 3: E p a r a is committed to serving real-time categorized AI parallel inference in edge clouds.
  • Figure 4: E p a r a system overview.
  • Figure 5: E p a r a task-categorized allocation.
  • ...and 15 more figures

Theorems & Definitions (2)

  • theorem 1: Submodular Function
  • theorem 2: Lower Bound Approximation