Sizey: Memory-Efficient Execution of Scientific Workflow Tasks

TL;DR

Sizey addresses the problem of memory wastage and underprovisioning in scientific workflows by offering an online memory-prediction framework that maintains multiple lightweight models per task-machine configuration. It uses a Resource Allocation Quality (RAQ) score to dynamically select or weight model predictions via Argmax or softmax-based Interpolation, and applies an adaptive offset to mitigate underestimation failures, with continuous online learning from completed tasks. The approach demonstrates a median memory-wastage reduction of at least 24.68% across six nf-core Nextflow workflows, outperforming four state-of-the-art baselines, and provides fast training (full retrain ~1.09s; incremental ~17.5ms) while preserving practical runtime efficiency. This work enables more efficient utilization of clusters by reducing memory over-provisioning without compromising workflow reliability, and it offers an open-source prototype for integration with existing SWMS and schedulers.

Abstract

As the amount of available data continues to grow in fields as diverse as bioinformatics, physics, and remote sensing, the importance of scientific workflows in the design and implementation of reproducible data analysis pipelines increases. When developing workflows, resource requirements must be defined for each type of task in the workflow. Typically, task types vary widely in their computational demands because they are simply wrappers for arbitrary black-box analysis tools. Furthermore, the resource consumption for the same task type can vary considerably as well due to different inputs. Since underestimating memory resources leads to bottlenecks and task failures, workflow developers tend to overestimate memory resources. However, overprovisioning of memory wastes resources and limits cluster throughput. Addressing this problem, we propose Sizey, a novel online memory prediction method for workflow tasks. During workflow execution, Sizey simultaneously trains multiple machine learning models and then dynamically selects the best model for each workflow task. To evaluate the quality of the model, we introduce a novel resource allocation quality (RAQ) score based on memory prediction accuracy and efficiency. Sizey's prediction models are retrained and re-evaluated online during workflow execution, continuously incorporating metrics from completed tasks. Our evaluation with a prototype implementation of Sizey uses metrics from six real-world scientific workflows from the popular nf-core framework and shows a median reduction in memory waste over time of 24.68% compared to the respective best-performing state-of-the-art baseline.

Figures (12)

• Figure 1: Distribution of peak memory consumption of four task types, each executed repeatedly with varying input sizes.
• Figure 2: Memory consumption in relation to the input read of the physical task instances of two different task types, with a linear regression applied to these data points.
• Figure 3: The figure provides an overview of how Sizey (green) is used in a scientific workflow execution environment. Sizey uses data from a provenance database and trains multiple machine learning models in parallel. The individual predictions are then combined into a single prediction that also includes an offset to ensure sufficient memory is allocated for tasks executing on the cluster. During workflow execution and after task completion, the model is retrained and reevaluated.
• Figure 4: Granularity levels for a given workflow with model A, task types 1, 2 on machines I and II; Sizey method in bold green box.
• Figure 5: The four model classes used by Sizey for modeling linear and non-linear relationships between input data sizes and peak memory consumption.