An Incremental Multi-Level, Multi-Scale Approach to Assessment of Multifidelity HPC Systems

TL;DR

The paper tackles the challenge of real-time analysis of terabyte-scale, multifidelity HPC logs by introducing an online, incremental multiresolution dynamic mode decomposition (I-mrDMD). It combines a streaming mrDMD framework with frequency-aware spectral isolation and a D3-based rack visualization in Jupyter to align environment, hardware, and job logs across multiple scales. The key contributions include the incremental SVD-based updates for multi-level mrDMD, a spectral mechanism for isolating significant modes, and an interactive visualization pipeline demonstrated on the Theta supercomputer with two case studies. This approach enables fast, scalable pattern discovery and anomaly detection in large-scale HPC systems, with practical implications for resilience, debugging, and efficient resource use, and is extensible to exascale and zettascale environments.

Abstract

With the growing complexity in architecture and the size of large-scale computing systems, monitoring and analyzing system behavior and events has become daunting. Monitoring data amounting to terabytes per day are collected by sensors housed in these massive systems at multiple fidelity levels and varying temporal resolutions. In this work, we develop an incremental version of multiresolution dynamic mode decomposition (mrDMD), which converts high-dimensional data to spatial-temporal patterns at varied frequency ranges. Our incremental implementation of the mrDMD algorithm (I-mrDMD) promptly reveals valuable information in the massive environment log dataset, which is then visually aligned with the processed hardware and job log datasets through our generalizable rack visualization using D3 visualization integrated into the Jupyter Notebook interface. We demonstrate the efficacy of our approach with two use scenarios on a real-world dataset from a Cray XC40 supercomputer, Theta.

Figures (9)

• Figure 1: Figure shows (a) Multiresolution Dynamic Mode Decomposition (mrDMD), (b) frequency Isolation of spatiotemporal modes using mrDMD spectrum, and (c) Incremental Multiresolution Dynamic Mode Decomposition (I-mrDMD)
• Figure 2: Generalizable rack layout view using D3 Visualization in Jupyter Notebook interface for the supercomputer Polaris polaris
• Figure 3: Analyzing the supercomputer environment logs containing baseline readings and non-baseline readings. (a) and (b) show the the actual data and the data reconstructed using the I-mrDMD modes, respectively
• Figure 4: Figure shows the Theta supercomputer ThetaANL rack (or node layout) view as described in Sec. \ref{['sec:rack_vis']}. The highlighted nodes are utilized by a job. The green hues represent reading closer to baseline readings, red hues represent high temperatures (much higher than baselines), and the blue hues represent low temperatures (much lower than baselines).
• Figure 5: Figure shows mode amplitude as a function of frequency (Eq. \ref{['freq_spec']}).The mrDMD spectrum shows dynamics of the data at different frequencies.