Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Machine Learning-Driven Predictive Resource Management in Complex Science Workflows

Tasnuva Chowdhury, Tadashi Maeno, Fatih Furkan Akman, Joseph Boudreau, Sankha Dutta, Shengyu Feng, Adolfy Hoisie, Kuan-Chieh Hsu, Raees Khan, Jaehyung Kim, Ozgur O. Kilic, Scott Klasky, Alexei Klimentov, Tatiana Korchuganova, Verena Ingrid Martinez Outschoorn, Paul Nilsson, David K. Park, Norbert Podhorszki, Yihui Ren, John Rembrandt Steele, Frédéric Suter, Sairam Sri Vatsavai, Torre Wenaus, Wei Yang, Yiming Yang, Shinjae Yoo

TL;DR

The paper tackles the challenge of predicting per-job resource needs in large-scale, heterogeneous scientific workflows managed by PanDA. It introduces a predictive pipeline combining a gradient-boosted baseline and deep neural networks with adaptive categorical embeddings to forecast RAM, CPU time, I/O intensity, and walltime as discrete classes, using four years of PanDA data. The ML pipeline is integrated into PanDA brokerage as containerized microservices and demonstrated to deliver sub-second inferences with strong end-to-end accuracy, dramatically surpassing the previous Scout-based two-stage approach. This work shows substantial potential to improve scheduling efficiency, reduce failures, and enable real-time, resource-aware scheduling across diverse computing environments.

Abstract

The collaborative efforts of large communities in science experiments, often comprising thousands of global members, reflect a monumental commitment to exploration and discovery. Recently, advanced and complex data processing has gained increasing importance in science experiments. Data processing workflows typically consist of multiple intricate steps, and the precise specification of resource requirements is crucial for each step to allocate optimal resources for effective processing. Estimating resource requirements in advance is challenging due to a wide range of analysis scenarios, varying skill levels among community members, and the continuously increasing spectrum of computing options. One practical approach to mitigate these challenges involves initially processing a subset of each step to measure precise resource utilization from actual processing profiles before completing the entire step. While this two-staged approach enables processing on optimal resources for most of the workflow, it has drawbacks such as initial inaccuracies leading to potential failures and suboptimal resource usage, along with overhead from waiting for initial processing completion, which is critical for fast-turnaround analyses. In this context, our study introduces a novel pipeline of machine learning models within a comprehensive workflow management system, the Production and Distributed Analysis (PanDA) system. These models employ advanced machine learning techniques to predict key resource requirements, overcoming challenges posed by limited upfront knowledge of characteristics at each step. Accurate forecasts of resource requirements enable informed and proactive decision-making in workflow management, enhancing the efficiency of handling diverse, complex workflows across heterogeneous resources.

Machine Learning-Driven Predictive Resource Management in Complex Science Workflows

TL;DR

The paper tackles the challenge of predicting per-job resource needs in large-scale, heterogeneous scientific workflows managed by PanDA. It introduces a predictive pipeline combining a gradient-boosted baseline and deep neural networks with adaptive categorical embeddings to forecast RAM, CPU time, I/O intensity, and walltime as discrete classes, using four years of PanDA data. The ML pipeline is integrated into PanDA brokerage as containerized microservices and demonstrated to deliver sub-second inferences with strong end-to-end accuracy, dramatically surpassing the previous Scout-based two-stage approach. This work shows substantial potential to improve scheduling efficiency, reduce failures, and enable real-time, resource-aware scheduling across diverse computing environments.

Abstract

The collaborative efforts of large communities in science experiments, often comprising thousands of global members, reflect a monumental commitment to exploration and discovery. Recently, advanced and complex data processing has gained increasing importance in science experiments. Data processing workflows typically consist of multiple intricate steps, and the precise specification of resource requirements is crucial for each step to allocate optimal resources for effective processing. Estimating resource requirements in advance is challenging due to a wide range of analysis scenarios, varying skill levels among community members, and the continuously increasing spectrum of computing options. One practical approach to mitigate these challenges involves initially processing a subset of each step to measure precise resource utilization from actual processing profiles before completing the entire step. While this two-staged approach enables processing on optimal resources for most of the workflow, it has drawbacks such as initial inaccuracies leading to potential failures and suboptimal resource usage, along with overhead from waiting for initial processing completion, which is critical for fast-turnaround analyses. In this context, our study introduces a novel pipeline of machine learning models within a comprehensive workflow management system, the Production and Distributed Analysis (PanDA) system. These models employ advanced machine learning techniques to predict key resource requirements, overcoming challenges posed by limited upfront knowledge of characteristics at each step. Accurate forecasts of resource requirements enable informed and proactive decision-making in workflow management, enhancing the efficiency of handling diverse, complex workflows across heterogeneous resources.

Paper Structure

This paper contains 12 sections, 4 equations, 6 figures, 5 tables.

Table of Contents

  1. Introduction
  2. PanDA Workflow Management System Overview
  3. Challenges in Resource Estimation and Motivation for Machine Learning
  4. Machine Learning-Based Predictive Resource Management
  5. Data, Features, Target Variables, and Preprocessing
  6. Model Architectures: Empirical Baseline and Deep Learning
  7. Model Training and Hyperparameter Tuning
  8. Evaluation Metrics and Model Performance
  9. System Integration and Workflow Impact
  10. Integrated Pipeline Evaluation for PanDA Brokerage
  11. Performance Benchmark and Workflow Impact
  12. Conclusion

Figures (6)

  • Figure 1: Representation of a basic workflow example composed of a sequence of workloads that depend on the previous step panda. Each workload is divided into jobs, which process an input data subset (e.g. a set of input files) and generate an output data subset (e.g. one output file). The output data of one workload is fed as input into the next workload in the sequence.
  • Figure 2: Completion of scout jobs per Task.
  • Figure 3: Distributions of key resource metrics across all tasks: (a) RAM usage, (b) CPU time, (c) I/O intensity, and (d) Walltime. Both axes are log-scaled to highlight the heavy-tailed nature of these resource requirements.
  • Figure 4: Distributions of discretized resource classes: (a) RAM Count, (b) CPU Time, (c) I/O Intensity and (d) Walltime. The class-based representation enables robust multi-class classification and aligns with operational resource allocation strategies.
  • Figure 5: Receiver Operating Characteristic (ROC) curves for all models. (a) Model 1: Multi-class ROC curves for RAM classification. (b) Model 2: Multi-class ROC curves for CPU time classification. (c) Model 3: Binary ROC curve for I/O intensity classification. (d) Model 4: Multi-class ROC curves for walltime classification. Micro-average AUC is shown for multiclass models.
  • ...and 1 more figures