Decentralized Distributed Proximal Policy Optimization (DD-PPO) for High Performance Computing Scheduling on Multi-User Systems

TL;DR

The paper tackles HPC batch scheduling by addressing multiple performance goals with a scalable reinforcement learning approach. It introduces DD-PPO, a decentralized distributed Proximal Policy Optimization algorithm, enabling large-scale training across many workers without centralized policy synchronization. Trained on over 11.5 million real HPC job traces and evaluated against rule-based and PPO baselines, DD-PPO demonstrates superior performance on several scheduling objectives and exhibits strong generalization to unseen traces. The work highlights the practical impact of scalable, data-rich RL for multi-user HPC systems and points to future exploration of applying DD-PPO to other RL schedulers.

Abstract

Resource allocation in High Performance Computing (HPC) environments presents a complex and multifaceted challenge for job scheduling algorithms. Beyond the efficient allocation of system resources, schedulers must account for and optimize multiple performance metrics, including job wait time and system utilization. While traditional rule-based scheduling algorithms dominate the current deployments of HPC systems, the increasing heterogeneity and scale of those systems is expected to challenge the efficiency and flexibility of those algorithms in minimizing job wait time and maximizing utilization. Recent research efforts have focused on leveraging advancements in Reinforcement Learning (RL) to develop more adaptable and intelligent scheduling strategies. Recent RL-based scheduling approaches have explored a range of algorithms, from Deep Q-Networks (DQN) to Proximal Policy Optimization (PPO), and more recently, hybrid methods that integrate Graph Neural Networks with RL techniques. However, a common limitation across these methods is their reliance on relatively small datasets, and these methods face scalability issues when using large datasets. This study introduces a novel RL-based scheduler utilizing the Decentralized Distributed Proximal Policy Optimization (DD-PPO) algorithm, which supports large-scale distributed training across multiple workers without requiring parameter synchronization at every step. By eliminating reliance on centralized updates to a shared policy, the DD-PPO scheduler enhances scalability, training efficiency, and sample utilization. The validation dataset leveraged over 11.5 million real HPC job traces for comparing DD-PPO performance between traditional and advanced scheduling approaches, and the experimental results demonstrate improved scheduling performance in comparison to both rule-based schedulers and existing RL-based scheduling algorithms.

Figures (9)

• Figure 1: Diagram showing the general framework of reinforcement learning.
• Figure 2: Overall architecture of our approach. Left box: The observable jobs are updated once an action is performed in the environment. Middle box: These jobs are sent as inputs into the RL agent policy and value networks. Right box: The agent next performs an action, which in turn updates the state and the environment, which returns a reward to the agent based on the optimization goal.
• Figure 3: Characteristics of the Falcon and Lemhi workload traces.
• Figure 4: Flow of model training and hyperparameter tuning using the Ray framework.
• Figure 5: Comparison of the proposed DD-PPO algorithm against several rule-based methods and the PPO algorithm using the Lublin-256 dataset (averaged across 10 runs).