Accurate and Scalable Many-Node Simulation

TL;DR

This work tackles the challenge of accurately projecting performance for future large-scale HPC systems by introducing a three-phase framework that blends native profiling, detailed node simulation, and high-level network simulation. By modeling timing variability, external communication, and problem-size scaling, the method achieves significantly lower prediction errors than prior approaches, exemplified by an average 6.7% error on 64 nodes across multiple benchmarks. The approach uses native profiling to capture rank divergence, detailed per-node simulation to obtain timing distributions, and skeleton-based network simulation to scale to thousands of nodes efficiently. This enables rapid exploration of design spaces, including node microarchitecture and network topology, with practical estimates obtainable within a day, facilitating better procurement and architecture decisions.

Abstract

Accurate performance estimation of future many-node machines is challenging because it requires detailed simulation models of both node and network. However, simulating the full system in detail is unfeasible in terms of compute and memory resources. State-of-the-art techniques use a two-phase approach that combines detailed simulation of a single node with network-only simulation of the full system. We show that these techniques, where the detailed node simulation is done in isolation, are inaccurate because they ignore two important node-level effects: compute time variability, and inter-node communication. We propose a novel three-stage simulation method to allow scalable and accurate many-node simulation, combining native profiling, detailed node simulation and high-level network simulation. By including timing variability and the impact of external nodes, our method leads to more accurate estimates. We validate our technique against measurements on a multi-node cluster, and report an average 6.7% error on 64 nodes (maximum error of 12%), compared to on average 27% error and up to 54% when timing variability and the scaling overhead are ignored. At higher node counts, the prediction error of ignoring variable timings and scaling overhead continues to increase compared to our technique, and may lead to selecting the wrong optimal cluster configuration. Using our technique, we are able to accurately project performance to thousands of nodes within a day of simulation time, using only a single or a few simulation hosts. Our method can be used to quickly explore large many-node design spaces, including node micro-architecture, node count and network configuration.

Figures (12)

• Figure 1: The many-node simulation methodology consists of three distinct phases.
• Figure 2: Illustration of a collective operation after a compute phase, using constant and variable time modeling. Variable timings more realistically model synchronization delays and network injection patterns, avoiding artificial traffic spikes.
• Figure 3: Node simulation with external ranks: 4 ranks are simulated in detail, the other ranks run natively on a different host.
• Figure 4: Measured and estimated execution times when scaling up to 64 nodes.
• Figure 5: Measured and estimated execution times when scaling up to 64 nodes.