Performance Analysis of HPC applications on the Aurora Supercomputer: Exploring the Impact of HBM-Enabled Intel Xeon Max CPUs

TL;DR

The paper investigates memory system design for the Aurora exascale system, emphasizing trade-offs between HBM and DDR memory and the impact of memory modes (Flat vs Cache) and clustering modes (Quad vs SNC4) on system and HPC application performance. It employs microbenchmarks (STREAM, Intel Latency Checker, OSU MPI benchmarks) and application trials on HACC, QMCPACK, and BFS to quantify bandwidth, latency, PCIe bandwidth, and MPI performance across configurations. Key findings show that HBM, particularly in SNC4-Flat mode, delivers higher memory bandwidth and lower latency, benefiting memory-bandwidth-bound workloads like HACC, while Cache mode with DDR capacity often helps latency-sensitive or larger-footprint workloads (QMCPACK, BFS) by leveraging DDR capacity; SNC4 requires careful process and NIC affinity to avoid imbalances. The work provides actionable guidance for selecting memory configurations on Aurora-like systems, highlighting the nuanced interplay between memory bandwidth, latency, and memory capacity across diverse HPC workloads and the importance of workload-aware tuning for exascale systems.

Abstract

The Aurora supercomputer is an exascale-class system designed to tackle some of the most demanding computational workloads. Equipped with both High Bandwidth Memory (HBM) and DDR memory, it provides unique trade-offs in performance, latency, and capacity. This paper presents a comprehensive analysis of the memory systems on the Aurora supercomputer, with a focus on evaluating the trade-offs between HBM and DDR memory systems. We explore how different memory configurations, including memory modes (Flat and Cache) and clustering modes (Quad and SNC4), influence key system performance metrics such as memory bandwidth, latency, CPU-GPU PCIe bandwidth, and MPI communication bandwidth. Additionally, we examine the performance of three representative HPC applications -- HACC, QMCPACK, and BFS -- each illustrating the impact of memory configurations on performance. By using microbenchmarks and application-level analysis, we provide insights into how to select the optimal memory system and configuration to maximize performance based on the application characteristics. The findings presented in this paper offer guidance for users of the Aurora system and similar exascale systems.

Figures (12)

• Figure 1: Block diagram of the Aurora Exascale Compute Blade in SNC4 mode, where each CPU socket is divided into four quadrants. The blade features six Ponte Vecchio (PVC) GPUs and a dual-socket Sapphire Rapids (SPR) CPU interconnected via three UPI links. Each CPU socket connects to three GPUs and four NICs through x16 PCIe Gen5 links.
• Figure 2: Memory modes x Clustering modes intel_xeon_max_guide.
• Figure 3: Single-socket memory bandwidth and loaded latency.
• Figure 4: CPU-GPU PCIe bandwidth across different memory and clustering modes.
• Figure 5: GPU PCIe bandwidth in SNC4-Flat/HBM mode with two different process placement strategies.