Scientific Workflow Scheduling in Cloud Considering Cold Start and Variable Pricing Model

TL;DR

This work tackles profitability for Scientific Cloud Service Providers by jointly addressing cold starts and the use of diverse VM pricing models in cloud-based scientific workflows. It introduces a two-phase hybrid scheduling framework that first plans with historical data (reserved/spot) and then adapts in real time with on-demand or spot provisioning, while accounting for task dependencies and deadlines. The Deadline, Cold Start, and Dependency-Aware (DCD) framework uses relative deadlines, a priority-driven VM selection, and a reward-guided spot bidding policy to balance performance and cost. Experiments with real workflows and pricing traces show that the approach outperforms state-of-the-art baselines under varying workload, spot availability, and prediction uncertainty, achieving significant profitability gains and robustness.

Abstract

Cloud computing has become a pivotal platform for executing scientific workflows due to its scalable and cost-effective infrastructure. Scientific Cloud Service Providers (SCSPs) act as intermediaries that rent virtual machines (VMs) from Infrastructure-as-a-Service (IaaS) providers to meet users' workflow execution demands. The SCSP earns profit from the execution of scientific workflows if it completes the execution of the workflow before the specified deadline of the workflow. This paper addresses two key challenges that impact the profitability of SCSPs: the cold start problem and the efficient management of diverse VM pricing models, namely reserved, on-demand, and spot instances. We propose a hybrid scheduling framework that integrates initial planning based on historical data with real-time adaptations informed by actual workload variations. In the initial phase, VMs are provisioned using reserved pricing based on predicted workloads and spot instances. During execution, the system dynamically adjusts by provisioning additional VMs through on-demand or spot instances to accommodate unexpected bursts in task arrivals. Our framework also incorporates a dependency-aware task scheduling strategy that accounts for cold start delays and spot pricing volatility. Experimental results on real-world benchmark datasets demonstrate that our approach outperforms state-of-the-art methods, achieving up to 20% improvement over cold-start-focused techniques and 15% over pricing-model-based VM provisioning strategies.

Figures (10)

• Figure 1: Considered System Architecture
• Figure 2: Temporal Pipeline of Proposed Approach. $X_{b}$, $Y_{b}$, $Z_{b}$, $A_{b}$ and $P_{b}$ represent predicted values, which are close but not identical to the actual values $X'_{tb}$, $Y'_{tb}$, $Z'_{b}$, $A'_{b}$, and $P'_{b}$. $b$ represents the corresponding batch number. The approach leveraging predicted workflows and spot instances operates ahead of time based on forecasted data, whereas the approach utilizing actual workflow data executes in real time. The trigger point of our proposed approach is highlighted by H.
• Figure 3: Flowchart of the Proposed Solution Approach for each batch. The numbers represent the ordering of the modules.
• Figure 4: A sample example showing the relative deadline calculation of all tasks in a workflow using \ref{['eqn:reldeadline']}. The critical path of workflow $W^k$ is marked inside the dotted line.
• Figure 5: Impact of Workflow Scaling on Cold Start and Deadline-Aware Scheduling Approaches