KS+: Predicting Workflow Task Memory Usage Over Time

TL;DR

KS+, a method that predicts a task’s memory consumption over time depending on its inputs, is proposed that dynamically segments the task execution and predicts the memory required for each segment.

Abstract

Scientific workflow management systems enable the reproducible execution of data analysis pipelines on cluster infrastructures managed by resource managers such as Kubernetes, Slurm, or HTCondor. These resource managers require resource estimates for each workflow task to be executed on one of the cluster nodes. However, task resource consumption varies significantly between different tasks and for the same task with different inputs. Furthermore, resource consumption also fluctuates during a task's execution. As a result, manually configuring static memory allocations is error-prone, often leading users to overestimate memory usage to avoid costly failures from under-provisioning, which results in significant memory wastage. We propose KS+, a method that predicts a task's memory consumption over time depending on its inputs. For this, KS+ dynamically segments the task execution and predicts the memory required for each segment. Our experimental evaluation shows an average reduction in memory wastage of 38% compared to the best-performing state-of-the-art baseline for two real-world workflows from the popular nf-core repository.

Figures (8)

• Figure 1: Variability of memory consumption a) for the Burrows-Wheeler Aligner (BWA) task with different inputs and b) for a single BWA task over time during execution.
• Figure 2: Memory consumption over time for a single execution of the BWA task. The dashed blue line represents the prediction with two uniform segments. The green line uses two segments with variable size, generated with our KS+ method.
• Figure 3: Start time for the second segment for all BWA task executions. The blue points represent individual task executions, the green line is the estimation, minimizing the mean squared error. The deviation gets larger for bigger input sizes. The red cross marks an execution that ran much faster than expected.
• Figure 4: Memory consumption over time of the BWA task execution marked in Figure \ref{['fig:segment_prediction']}. The dashed blue line represents the predicted KS+ allocation strategy, that fails due to the execution reaching the second segment faster than expected. The green line shows the allocation for the second try.
• Figure 5: Memory consumption of the experimental workflow tasks.