Comparing Cross-Platform Performance via Node-to-Node Scaling Studies

TL;DR

Cross-platform HPC performance comparisons face practical setup challenges across heterogeneous architectures. The paper proposes node-to-node scaling as the core unit of comparison and provides plotting templates for strong, weak, strong-weak, and throughput analyses, along with an end-to-end workflow using Maestro, Caliper/Adiak, Thicket, and pandas. It contributes a structured visualization toolkit and multiple Marbl-based case studies across CPU, GPU, ARM, and cloud environments, demonstrating insights such as memory-bandwidth saturation and the benefits of shared memory pools and matrix-free remap. The work enables more informed platform selection, procurement decisions, and performance-portable code design, while acknowledging data-tracking burdens and opportunities for automation in future work. Key metrics include the strong scaling speedup $t_{P}(1)/t_{P}(N)$, the weak scaling efficiency $t_{P}(1)/t_{P}(N)$, and throughput defined as $throughput = \frac{\langle DOFs-processed\rangle}{\text{compute\_node}} \cdot \frac{cycles}{second}$, with charts employing $log_2$ and $log_{10}$ scales to visualize scaling behavior.

Abstract

Due to the increasing diversity of high-performance computing architectures, researchers and practitioners are increasingly interested in comparing a code's performance and scalability across different platforms. However, there is a lack of available guidance on how to actually set up and analyze such cross-platform studies. In this paper, we contend that the natural base unit of computing for such studies is a single compute node on each platform and offer guidance in setting up, running, and analyzing node-to-node scaling studies. We propose templates for presenting scaling results of these studies and provide several case studies highlighting the benefits of this approach.

Figures (15)

• Figure 1: Proposed chart for node-to-node strong scaling studies plotting time per cycle (in seconds) against the number of compute nodes. Under ideal strong scaling, runtime decreases proportionally with the number of compute nodes, with a slope of $-1$ on the $log_2$--$log_2$ axes.
• Figure 2: Proposed chart for node-to-node weak scaling studies plotting time per cycle (in seconds) against the number of compute nodes. Under ideal weak scaling, runtime is independent of the number of compute nodes, yielding a horizontal line of slope $0$. Each horizontal bar corresponds to a factor of two increase in runtime.
• Figure 3: Proposed chart for node-to-node strong-weak studies plotting time per cycle (in seconds) against the number of compute nodes. By carefully controlling the study parameters, we can include both strong scaling and weak scaling on the same chart. Solid diagonal lines correspond to strong scaling. Dashed horizontal lines correspond to weak scaling.
• Figure 4: Proposed chart for node-to-node throughput scaling studies plotting throughput against the problem size (that is, degrees of freedom). Throughput helps us understand how well a code is using its compute resources.
• Figure 5: We set up our scaling study parameters and machine details in templated file that are processed into YAML files runnable by Maestro.