Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SCAR: Scheduling Multi-Model AI Workloads on Heterogeneous Multi-Chiplet Module Accelerators

Mohanad Odema, Luke Chen, Hyoukjun Kwon, Mohammad Abdullah Al Faruque

TL;DR

This work develops a set of heuristics to navigate the huge scheduling space and codify them into a scheduler, SCAR, with advanced techniques such as inter-chiplet pipelining, and evaluation on ten multi-model workload scenarios has shown the efficacy of this approach.

Abstract

Emerging multi-model workloads with heavy models like recent large language models significantly increased the compute and memory demands on hardware. To address such increasing demands, designing a scalable hardware architecture became a key problem. Among recent solutions, the 2.5D silicon interposer multi-chip module (MCM)-based AI accelerator has been actively explored as a promising scalable solution due to their significant benefits in the low engineering cost and composability. However, previous MCM accelerators are based on homogeneous architectures with fixed dataflow, which encounter major challenges from highly heterogeneous multi-model workloads due to their limited workload adaptivity. Therefore, in this work, we explore the opportunity in the heterogeneous dataflow MCM AI accelerators. We identify the scheduling of multi-model workload on heterogeneous dataflow MCM AI accelerator is an important and challenging problem due to its significance and scale, which reaches O(10^56) even for a two-model workload on 6x6 chiplets. We develop a set of heuristics to navigate the huge scheduling space and codify them into a scheduler, SCAR, with advanced techniques such as inter-chiplet pipelining. Our evaluation on ten multi-model workload scenarios for datacenter multitenancy and AR/VR use-cases has shown the efficacy of our approach, achieving on average 27.6% and 29.6% less energy-delay product (EDP) for the respective applications settings compared to homogeneous baselines.

SCAR: Scheduling Multi-Model AI Workloads on Heterogeneous Multi-Chiplet Module Accelerators

TL;DR

This work develops a set of heuristics to navigate the huge scheduling space and codify them into a scheduler, SCAR, with advanced techniques such as inter-chiplet pipelining, and evaluation on ten multi-model workload scenarios has shown the efficacy of this approach.

Abstract

Emerging multi-model workloads with heavy models like recent large language models significantly increased the compute and memory demands on hardware. To address such increasing demands, designing a scalable hardware architecture became a key problem. Among recent solutions, the 2.5D silicon interposer multi-chip module (MCM)-based AI accelerator has been actively explored as a promising scalable solution due to their significant benefits in the low engineering cost and composability. However, previous MCM accelerators are based on homogeneous architectures with fixed dataflow, which encounter major challenges from highly heterogeneous multi-model workloads due to their limited workload adaptivity. Therefore, in this work, we explore the opportunity in the heterogeneous dataflow MCM AI accelerators. We identify the scheduling of multi-model workload on heterogeneous dataflow MCM AI accelerator is an important and challenging problem due to its significance and scale, which reaches O(10^56) even for a two-model workload on 6x6 chiplets. We develop a set of heuristics to navigate the huge scheduling space and codify them into a scheduler, SCAR, with advanced techniques such as inter-chiplet pipelining. Our evaluation on ten multi-model workload scenarios for datacenter multitenancy and AR/VR use-cases has shown the efficacy of our approach, achieving on average 27.6% and 29.6% less energy-delay product (EDP) for the respective applications settings compared to homogeneous baselines.
Paper Structure (29 sections, 2 theorems, 20 equations, 13 figures, 7 tables, 1 algorithm)

This paper contains 29 sections, 2 theorems, 20 equations, 13 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Multi-model AI Workloads
  4. Scheduling AI workloads on AI Hardware and MCMs
  5. Motivational Example
  6. Complexity of the multi-model scheduling space
  7. Summary on the challenges of multi-model scheduling
  8. System Modeling and Problem Formulation
  9. Base Formulation
  10. Workload Partitioning Space
  11. Scheduling Space
  12. Scheduling Problem
  13. Performance Modeling
  14. Proposed Scheduling Framework: SCAR
  15. MCM Reconfiguration Engine (MCM-Reconfig)
  16. ...and 14 more sections

Key Result

Theorem 1

The validity of segments in a time window For a time window $tw(Sc)$ and its layers $L(tw(Sc))$, let the set of all segments for $tw(Sc)$ be $SG$, then $SG$ is valid if the following condition is satisfied:

Figures (13)

  • Figure 1: An overview of this work: (a) Emerging Multi-model workloads have introduced new challenges for AI hardware. (b) Heterogeneous MCMs present a promising solution to scale with multi-model workloads with some considerations. (c) Our proposed scheduling framework addresses said challenges to explore the heterogeneous scheduling space. (d) Provided Solutions provide optimized spatio-temporal scheduling strategies for the multi-model workloads.
  • Figure 2: Motivational study on a $2\times2$ MCM AI accelerator using batch size of 1 for 3 layers from the second ResNet-50 block and the first feed forward layer from GPT-2. Each chiplet has 4096 PEs and 10 MB L2 shared memory (Full from Section \ref{['sec:eval']}). NN-baton tan2021nn considers partitioning computation across chiplets only when not enough resources exist.
  • Figure 3: An overview of our scheduling algorithm for multi-model workloads on heterogeneous MCM AI accelerators.
  • Figure 4: Our proposed multi-model scheduling framework on heterog. MCM.
  • Figure 5: Schedules creation through the SEG and SCHED engines.
  • ...and 8 more figures

Theorems & Definitions (12)

  • Definition 1
  • Definition 2
  • Definition 3
  • Definition 4
  • Definition 5
  • Theorem 1
  • Theorem 2
  • Definition 6
  • Definition 7
  • Definition 8
  • ...and 2 more