Workflow-Driven Modeling for the Compute Continuum: An Optimization Approach to Automated System and Workload Scheduling

TL;DR

The paper addresses the challenge of coordinating cloud and HPC resources for tightly coupled workflows in a compute continuum. It proposes a comprehensive framework based on rigorous system and workload modeling, integrating MILP and heuristic scheduling, extended Snakemake, and JSON-based inputs to automate mapping and scheduling across cloud and on-prem HPC resources. The authors show that a $MILP$-based solution achieves optimal scheduling and makespan for small-scale workflows, while heuristic methods yield up to $99\%$ faster estimations with a $5$–$10\%$ deviation for large-scale cases, validated on MRI-like workflows. Overall, the framework closes gaps in current tooling and enables fully automated orchestration in heterogeneous HPC-CC environments, with significant potential to improve resource utilization and execution efficiency.

Abstract

The convergence of IoT, Edge, Cloud, and HPC technologies creates a compute continuum that merges cloud scalability and flexibility with HPC's computational power and specialized optimizations. However, integrating cloud and HPC resources often introduces latency and communication overhead, which can hinder the performance of tightly coupled parallel applications. Additionally, achieving seamless interoperability between cloud and on-premises HPC systems requires advanced scheduling, resource management, and data transfer protocols. Consequently, users must manually allocate complex workloads across heterogeneous resources, leading to suboptimal task placement and reduced efficiency due to the absence of an automated scheduling mechanism. To overcome these challenges, we introduce a comprehensive framework based on rigorous system and workload modeling for the compute continuum. Our method employs established tools and techniques to optimize workload mapping and scheduling, enabling the automatic orchestration of tasks across both cloud and HPC infrastructures. Experimental evaluations reveal that our approach could optimally improve scheduling efficiency, reducing execution times, and enhancing resource utilization. Specifically, our MILP-based solution achieves optimal scheduling and makespan for small-scale workflows, while heuristic methods offer up to 99% faster estimations for large-scale workflows, albeit with a 5-10% deviation from optimal results. Our primary contribution is a robust system and workload modeling framework that addresses critical gaps in existing tools, paving the way for fully automated orchestration in HPC-compute continuum environments.

Figures (11)

• Figure 1: H-HPC-CC (Heterogeneous High-Performance Computing Compute Continuum): Real-world Scenario.
• Figure 2: Diagrams illustrating the MRI use case and corresponding workflows.
• Figure 3: Heterogeneous workflows on the Compute Continuum for different users.
• Figure 4: Workload mapping and scheduling framework with auto estimation of incoming workflows.
• Figure 5: Original Snakefile rule.