Novel Representation Learning Technique using Graphs for Performance Analytics

TL;DR

This work tackles HPC runtime prediction from tabular configurations by transforming tabular data into graphs where nodes are samples and edges encode sample similarities. A self-supervised edge-inference pipeline and GraphSAGE-based representation learning produce embeddings that enable a simple regressor to achieve superior MSE performance compared to DNNs and other baselines, even with up to 25% missing data. The approach is validated on 10 HPC datasets and 3 ML benchmarks, showing up to 61.67% and 78.56% MSE improvements over DNN baselines, and demonstrates robustness suitable for streaming scenarios. By explicitly modeling inter-sample relationships, the method provides scalable, explainable embeddings that improve regression tasks in performance analytics and beyond.

Abstract

The performance analytics domain in High Performance Computing (HPC) uses tabular data to solve regression problems, such as predicting the execution time. Existing Machine Learning (ML) techniques leverage the correlations among features given tabular datasets, not leveraging the relationships between samples directly. Moreover, since high-quality embeddings from raw features improve the fidelity of the downstream predictive models, existing methods rely on extensive feature engineering and pre-processing steps, costing time and manual effort. To fill these two gaps, we propose a novel idea of transforming tabular performance data into graphs to leverage the advancement of Graph Neural Network-based (GNN) techniques in capturing complex relationships between features and samples. In contrast to other ML application domains, such as social networks, the graph is not given; instead, we need to build it. To address this gap, we propose graph-building methods where nodes represent samples, and the edges are automatically inferred iteratively based on the similarity between the features in the samples. We evaluate the effectiveness of the generated embeddings from GNNs based on how well they make even a simple feed-forward neural network perform for regression tasks compared to other state-of-the-art representation learning techniques. Our evaluation demonstrates that even with up to 25% random missing values for each dataset, our method outperforms commonly used graph and Deep Neural Network (DNN)-based approaches and achieves up to 61.67% & 78.56% improvement in MSE loss over the DNN baseline respectively for HPC dataset and Machine Learning Datasets.

Figures (6)

• Figure 1: A sample performance dataset. It contains numerical and categorical features in the first six columns and the target value in the last (Runtime). Note that values colored in red (NaN) are missing values. Streaming performance monitoring systems often miss recording feature values or cannot measure the runtime until the application ends, resulting in missing feature values and unlabeled samples.
• Figure 2: Illustration of the Single-graph construction approach using $N$ Cosine similarity. Here, $N$ is a user-defined parameter that determines the density of the performance graph.
• Figure 3: Outcome of the batched-graph construction approach.
• Figure 4: The end-to-end pipeline of learning the optimized graph from initially constructed formulation. Input graph and its edge weights are fed to a model with self-supervision for constructing an embedding for the entire dataset. Here, solid lines represent forward propagation, and dotted lines represent backpropagation to enhance the edge weights.
• Figure 5: Normalized MSE Loss for HPC dataset The X-axis shows the % of missing feature values and the Y-axis shows the normalized MSE value for different methods.