Analytic Roofline Modeling and Energy Analysis of LULESH Proxy Application on Multi-Core Clusters
Ayesha Afzal, Georg Hager, Gerhard Wellein
TL;DR
This work applies analytic Roofline modeling to the LULESH proxy application implemented with OpenMP and MPI on two Intel multi-core clusters (Ice Lake and Sapphire Rapids) to quantify performance and energy performance. It develops predictive models for the memory-bound hot-spot kernels, validates them against hardware counters, and analyzes how problem size, core/uncore frequency, and programming model choices affect energy-to-solution and energy-delay product. The study shows that memory-bound kernels dominate energy use and that substantial energy savings are achievable by tuning core counts and frequencies, though very high baseline idle power on SPR can limit concurrency throttling benefits. Across single- and multi-node runs, MPI generally exhibits strong weak scaling, while OpenMP overheads are more impactful at small problem sizes; SPR often achieves lower EDP for memory-bound workloads thanks to higher memory bandwidth, yet total energy can be similar due to larger clock-to-power trade-offs. The methodology and insights are broadly applicable to memory-intensive HPC codes beyond LULESH, informing energy-aware performance optimization strategies.
Abstract
We present a thorough performance and energy consumption analysis of the LULESH proxy application in its OpenMP and MPI variants on two different clusters based on Intel Ice Lake (ICL) and Sapphire Rapids (SPR) CPUs. We first study the strong scaling and power consumption characteristics of the six hot spot functions in the code on the node level, with a special focus on memory bandwidth utilization. We then proceed with the construction of a detailed Roofline performance model for each memory-bound hot spot, which we validate using hardware performance counter measurements. We also comment on the observed discrepancies between the analytical model and the observations. To discern the influence of the programming model from the influence of implementation of the code, we compare the performance of OpenMP and MPI based on problem size, examining if the underlying implementation is equivalent for large problems, and if differences in overheads are more significant at smaller problem sizes. We also conduct an analysis of the power dissipation, energy to solution, and energy-delay product (EDP) of the hot spots, quantifying the influence of problem size, core and uncore clock frequency, and number of active cores per ccNUMA domain. Relevant energy savings are only possible for memory-bound functions by using fewer cores per ccNUMA domain and/or reducing the core clock speed. A major issue is the very high extrapolated baseline power on both chips, which makes concurrency throttling less effective. In terms of energy-delay product (EDP), on SPR only memory-bound workloads offer lower EDP compared to Ice Lake.