Characterizing GPU Energy Usage in Exascale-Ready Portable Science Applications

TL;DR

This work tackles the problem of understanding GPU energy usage in exascale-ready HPC applications to inform energy-aware hardware-software co-design. It introduces HWEnergyTracer.jl to collect power, temperature, and utilization traces via NVML and rocm_smi_lib, and applies it to QMCPACK and AMReX-Castro across NVIDIA A100/H100 and AMD MI250X GPUs, presenting energy traces and an energy-aware throughput metric. The key contributions include quantifying energy savings from mixed precision (approximately 6–25% for QMCPACK and up to 45% for AMReX-Castro on NVIDIA GPUs), proposing the metric $Throughput_{Energy} = \frac{walkers \times blocks \times steps}{DMC\ energy}$, and highlighting tooling gaps on AMD hardware that affect energy assessments. The findings underscore the importance of application-level energy analyses for exascale co-design and demonstrate that precision and hardware differences significantly shape energy efficiency, with H100 offering notable gains over A100 and substantial room for tooling improvements on AMD platforms.

Abstract

We characterize the GPU energy usage of two widely adopted exascale-ready applications representing two classes of particle and mesh solvers: (i) QMCPACK, a quantum Monte Carlo package, and (ii) AMReXCastro, an adaptive mesh astrophysical code. We analyze power, temperature, utilization, and energy traces from double-/single (mixed)-precision benchmarks on NVIDIA's A100 and H100 and AMD's MI250X GPUs using queries in NVML and rocm_smi_lib, respectively. We explore application-specific metrics to provide insights on energy vs. performance trade-offs. Our results suggest that mixed-precision energy savings range between 6-25% on QMCPACK and 45% on AMReX-Castro. Also, we found gaps in the AMD tooling used on Frontier GPUs that need to be understood, while query resolutions on NVML have little variability between 1 ms-1 s. Overall, application level knowledge is crucial to define energy-cost/science-benefit opportunities for the codesign of future supercomputer architectures in the post-Moore era.

Figures (8)

• Figure 1: QMCPACK NiO Benchmark DMC GPU traces on an NVIDIA H100.
• Figure 2: Energy characteristics of the QMCPACK NiO benchmark on NVIDIA H100 and A100 for different query time resolutions.
• Figure 3: Energy characteristics of the QMCPACK NiO benchmark on an AMD MI250X for different query time resolutions.
• Figure 4: Mixed-precision traces on NVIDIA H100 and A100 for (a) max double-precision walkers (68 and 58) and (b) max mixed-precision walkers (100 and 84).
• Figure 5: AMD MI250X mixed-precision traces for (a) max double-precision walkers (38) and (b) max mixed-precision walkers (52).