Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Orchestrated AI Workflows Execution on Scale-out Spatial Architecture

Jinyi Deng, Xinru Tang, Zhiheng Yue, Guangyang Lu, Qize Yang, Jiahao Zhang, Jinxi Li, Chao Li, Shaojun Wei, Yang Hu, Shouyi Yin

TL;DR

This work defines Orchestrated AI Workflows (OAW) and formalizes their Dual Dynamicity through the Orchestrated Workflow Graph (OWG), which captures how Task Blocks and Control Blocks interact through Task Flows. To address the two-level dynamicity, the authors propose Octopus, a scale-out spatial architecture with TBUs and CBUs and a decoupled control plane, equipped with Discriminate Dual-Scheduling, Adaptive TBU Scheduling, and Proactive Cluster Scheduling. Their cycle-level simulations and wafer-scale experiments show Octopus achieving an average 2.50× speedup (up to 4.26×) over a baseline and robust scalability, including a wafer-scale demonstration with diffusion-based load balancing. The results indicate that combining fine-grained TB scheduling with coarse-grained cluster-level planning yields significant gains in utilization and performance for dynamic Orchestrated AI Workloads, enabling practical deployment on large-scale hardware.

Abstract

Given the increasing complexity of AI applications, traditional spatial architectures frequently fall short. Our analysis identifies a pattern of interconnected, multi-faceted tasks encompassing both AI and general computational processes. In response, we have conceptualized "Orchestrated AI Workflows," an approach that integrates various tasks with logic-driven decisions into dynamic, sophisticated workflows. Specifically, we find that the intrinsic Dual Dynamicity of Orchestrated AI Workflows, namely dynamic execution times and frequencies of Task Blocks, can be effectively represented using the Orchestrated Workflow Graph. Furthermore, the intrinsic Dual Dynamicity poses challenges to existing spatial architecture, namely Indiscriminate Resource Allocation, Reactive Load Rebalancing, and Contagious PEA Idleness. To overcome these challenges, we present Octopus, a scale-out spatial architecture and a suite of advanced scheduling strategies optimized for executing Orchestrated AI Workflows, such as the Discriminate Dual-Scheduling Mechanism, Adaptive TBU Scheduling Strategy, and Proactive Cluster Scheduling Strategy. Our evaluations demonstrate that Octopus significantly outperforms traditional architectures in handling the dynamic demands of Orchestrated AI Workflows, and possesses robust scalability in large scale hardware such as wafer-scale chip.

Efficient Orchestrated AI Workflows Execution on Scale-out Spatial Architecture

TL;DR

This work defines Orchestrated AI Workflows (OAW) and formalizes their Dual Dynamicity through the Orchestrated Workflow Graph (OWG), which captures how Task Blocks and Control Blocks interact through Task Flows. To address the two-level dynamicity, the authors propose Octopus, a scale-out spatial architecture with TBUs and CBUs and a decoupled control plane, equipped with Discriminate Dual-Scheduling, Adaptive TBU Scheduling, and Proactive Cluster Scheduling. Their cycle-level simulations and wafer-scale experiments show Octopus achieving an average 2.50× speedup (up to 4.26×) over a baseline and robust scalability, including a wafer-scale demonstration with diffusion-based load balancing. The results indicate that combining fine-grained TB scheduling with coarse-grained cluster-level planning yields significant gains in utilization and performance for dynamic Orchestrated AI Workloads, enabling practical deployment on large-scale hardware.

Abstract

Given the increasing complexity of AI applications, traditional spatial architectures frequently fall short. Our analysis identifies a pattern of interconnected, multi-faceted tasks encompassing both AI and general computational processes. In response, we have conceptualized "Orchestrated AI Workflows," an approach that integrates various tasks with logic-driven decisions into dynamic, sophisticated workflows. Specifically, we find that the intrinsic Dual Dynamicity of Orchestrated AI Workflows, namely dynamic execution times and frequencies of Task Blocks, can be effectively represented using the Orchestrated Workflow Graph. Furthermore, the intrinsic Dual Dynamicity poses challenges to existing spatial architecture, namely Indiscriminate Resource Allocation, Reactive Load Rebalancing, and Contagious PEA Idleness. To overcome these challenges, we present Octopus, a scale-out spatial architecture and a suite of advanced scheduling strategies optimized for executing Orchestrated AI Workflows, such as the Discriminate Dual-Scheduling Mechanism, Adaptive TBU Scheduling Strategy, and Proactive Cluster Scheduling Strategy. Our evaluations demonstrate that Octopus significantly outperforms traditional architectures in handling the dynamic demands of Orchestrated AI Workflows, and possesses robust scalability in large scale hardware such as wafer-scale chip.
Paper Structure (24 sections, 15 figures, 2 tables)

This paper contains 24 sections, 15 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Workload Characterization
  3. Orchestrated AI Workflow
  4. Orchestrated Workflow Graph
  5. Spatial Architecture and Challenges
  6. Spatial Architecture
  7. Spatial Architecture Challenges
  8. Design
  9. Overall Architecture
  10. Discriminate Dual-Scheduling Mechanism
  11. Adaptive TBU Scheduling Strategy
  12. Proactive Cluster Scheduling Strategy
  13. Implementation
  14. Evaluation Methodology
  15. Comparison Methodology
  16. ...and 9 more sections

Figures (15)

  • Figure 1: Orchestrated AI Workflow Example: Emotion Recognition; Orchestrated Workflow Graph Extracting the Inherent Logic and intrinsic characteristics of Orchestrated AI Workflow.
  • Figure 2: Orchestrated Workflow Graph and Dual Dynamicity.
  • Figure 3: Basic Spatial Architecture and Limitations in Handling Orchestrated AI Workflows.
  • Figure 4: Challenges Issues faced by present spatial architectures in executing a dynamic schedule effectively while managing Orchestrated AI Workflows.
  • Figure 5: Overall design encompasses a control-flow plane architected spatial architecture, supported by a suite of software strategies. Each strategy is backed by the microarchitecture of the control flow plane within TBUs and CBUs.
  • ...and 10 more figures