Analytics of Longitudinal System Monitoring Data for Performance Prediction

TL;DR

This work tackles HPC performance variability by predicting the runtime of pending jobs from pre-execution system state. It leverages longitudinal LDMS-based measurements from Cori, constructs a scalable feature pipeline with per-router reductions and groupings, and trains GBR-based predictors complemented by a neural network for broader generalization. The results show strong, application-agnostic predictive power and identify key hardware counters that signal potential degradation, enabling lightweight, scheduler-aware decisions. The findings suggest practical pathways to reduce variability and improve throughput through pre-runtime monitoring and targeted resource-aware scheduling.

Abstract

In recent years, several HPC facilities have started continuous monitoring of their systems and jobs to collect performance-related data for understanding performance and operational efficiency. Such data can be used to optimize the performance of individual jobs and the overall system by creating data-driven models that can predict the performance of jobs waiting in the scheduler queue. In this paper, we model the performance of representative control jobs using longitudinal system-wide monitoring data and machine learning to explore the causes of performance variability. We analyze these prediction models in great detail to identify the features that are dominant predictors of performance. We demonstrate that such models can be application-agnostic and can be used for predicting performance of applications that are not included in training.

Figures (11)

• Figure 1: Variability in the performance of 128-node AMG and MILC jobs on different days (on Cori at NERSC).
• Figure 2: Network ports classified into router and processor tiles on the 48-port Aries router (left) and a multi-group dragonfly system constructed using the Aries router as a building block (right).
• Figure 3: LDMS data five minutes prior to job start is used as input to train the machine learning models.
• Figure 4: MAPE and PSLE scores for the GBR model when using different router types for filtering the system-wide data.
• Figure 5: Relative importances of the most important counters obtained using RFE for different datasets (aggregation function: mean).