Scalable HPC Job Scheduling and Resource Management in SST

TL;DR

The paper tackles the need for scalable, high-fidelity HPC job scheduling with workflow dependencies by embedding a scalable, SST-based scheduling component and a DAG-oriented workflow manager into the Structural Simulation Toolkit. It implements five scheduling algorithms (FCFS, FCFS with Backfilling, FCFS with Best Fit, Shortest Job First, and Longest Job First) and provides a workflow subsystem that handles task DAGs, dependencies, and resource allocations within SST's discrete-event framework. Validation against Grid Workloads Archive traces and CQSim demonstrates accurate wait times and node utilization, while scalability experiments show good parallel performance for both the scheduler and workflow components; simplified Pegasus/Airflow workflows further corroborate dependency handling. The work offers a practical, high-fidelity platform for evaluating HPC scheduling strategies and workflow dynamics, with planned extensions to dynamic scheduling, preemption, heterogeneous/multi-cluster operation, and deeper integration with established workflow systems.

Abstract

Efficient job scheduling and resource management contribute towards system throughput and efficiency maximization in high-performance computing (HPC) systems. In this paper, we introduce a scalable job scheduling and resource management component within the structural simulation toolkit (SST), a cycle-accurate and parallel discrete-event simulator. Our proposed simulator includes state-of-the-art job scheduling algorithms and resource management techniques. Additionally, it introduces workflow management components that support the simulation of task dependencies and resource allocations, crucial for workflows typical in scientific computing and data-intensive applications. We present the validation and scalability results of our job scheduling simulator. Simulation shows that our simulator achieves good accuracy in various metrics (e.g., job wait times, number of nodes usage) and also achieves good parallel performance.

Figures (7)

• Figure 1: The simulator components.
• Figure 2: The workflow management components.
• Figure 3: Comparison of our simulation output with CQSim.
• Figure 4: Job wait time validation and comparison of different scheduling algorithms.
• Figure 5: Scalability of our proposed simulator.