Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Predicting the Performance of Scientific Workflow Tasks for Cluster Resource Management: An Overview of the State of the Art

Jonathan Bader, Kathleen West, Soeren Becker, Svetlana Kulagina, Fabian Lehmann, Lauritz Thamsen, Henning Meyerhenke, Odej Kao

TL;DR

This chapter surveys state-of-the-art methods for predicting the runtime and memory usage of individual scientific workflow tasks to improve cluster resource management. It systematically classifies methods by four characteristics (General, Model, Model Input, Evaluation) and by execution phase (offline, online, pre-execution), contrasting nineteen papers and highlighting opportunities for real-world deployment. The discussion extends to applications in workflow scheduling, energy and carbon-aware execution, and cost optimization, emphasizing the benefits and limitations of current prediction approaches. Overall, the work provides a comprehensive blueprint of task-level performance prediction and points to key gaps, such as GPU-focused runtime prediction, memory-heterogeneity considerations, and the need for real-system evaluations and open-source implementations to enhance reproducibility and adoption.

Abstract

Scientific workflow management systems support large-scale data analysis on cluster infrastructures. For this, they interact with resource managers which schedule workflow tasks onto cluster nodes. In addition to workflow task descriptions, resource managers rely on task performance estimates such as main memory consumption and runtime to efficiently manage cluster resources. Such performance estimates should be automated, as user-based task performance estimates are error-prone. In this book chapter, we describe key characteristics of methods for workflow task runtime and memory prediction, provide an overview and a detailed comparison of state-of-the-art methods from the literature, and discuss how workflow task performance prediction is useful for scheduling, energy-efficient and carbon-aware computing, and cost prediction.

Predicting the Performance of Scientific Workflow Tasks for Cluster Resource Management: An Overview of the State of the Art

TL;DR

This chapter surveys state-of-the-art methods for predicting the runtime and memory usage of individual scientific workflow tasks to improve cluster resource management. It systematically classifies methods by four characteristics (General, Model, Model Input, Evaluation) and by execution phase (offline, online, pre-execution), contrasting nineteen papers and highlighting opportunities for real-world deployment. The discussion extends to applications in workflow scheduling, energy and carbon-aware execution, and cost optimization, emphasizing the benefits and limitations of current prediction approaches. Overall, the work provides a comprehensive blueprint of task-level performance prediction and points to key gaps, such as GPU-focused runtime prediction, memory-heterogeneity considerations, and the need for real-system evaluations and open-source implementations to enhance reproducibility and adoption.

Abstract

Scientific workflow management systems support large-scale data analysis on cluster infrastructures. For this, they interact with resource managers which schedule workflow tasks onto cluster nodes. In addition to workflow task descriptions, resource managers rely on task performance estimates such as main memory consumption and runtime to efficiently manage cluster resources. Such performance estimates should be automated, as user-based task performance estimates are error-prone. In this book chapter, we describe key characteristics of methods for workflow task runtime and memory prediction, provide an overview and a detailed comparison of state-of-the-art methods from the literature, and discuss how workflow task performance prediction is useful for scheduling, energy-efficient and carbon-aware computing, and cost prediction.
Paper Structure (33 sections, 2 figures)

This paper contains 33 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Workflow Task Runtime Prediction
  3. Characteristics
  4. General:
  5. Model:
  6. Model Input:
  7. Evaluation:
  8. Offline Workflow Task Runtime Prediction
  9. Online Workflow Task Runtime Prediction
  10. Pre-Execution Workflow Task Runtime Prediction
  11. Discussion
  12. Workflow Task Memory Prediction
  13. Characteristics
  14. General:
  15. Model:
  16. ...and 18 more sections

Figures (2)

  • Figure 1: Interaction between a scientific workflow management system, a resource manager, and task performance prediction models.
  • Figure 2: Typical execution environment where task performance predictions are used for scheduling, energy-efficient and carbon-aware execution, and cost prediction. The cluster can take different forms, such as an on-premise system, a virtual cluster in the cloud, or a hybrid solution.