OpenACC offloading of the MFC compressible multiphase flow solver on AMD and NVIDIA GPUs

TL;DR

The paper addresses the challenge of delivering portable, high-performance multiphase flow simulations on modern GPUs from multiple vendors. It adopts OpenACC directive offloading for the MFC solver, employing careful data-layout optimizations, loop restructuring, and vendor-library acceleration to achieve significant kernel speedups and robust scaling on NVIDIA and AMD platforms. Key contributions include a comprehensive optimization pipeline (gang/vector with collapse, array packing, inlining via Fypp), GEAM-based transposes via hipBLAS/cuTENSOR, and substantial weak/strong scaling gains, with benchmark results on Summit and Frontier and validation against canonical multiphase-flow problems. The practical impact is a portable, scalable framework enabling leadership-class simulations across diverse GPU architectures with competitive performance and portability.

Abstract

GPUs are the heart of the latest generations of supercomputers. We efficiently accelerate a compressible multiphase flow solver via OpenACC on NVIDIA and AMD Instinct GPUs. Optimization is accomplished by specifying the directive clauses 'gang vector' and 'collapse'. Further speedups of six and ten times are achieved by packing user-defined types into coalesced multidimensional arrays and manual inlining via metaprogramming. Additional optimizations yield seven-times speedup in array packing and thirty-times speedup of select kernels on Frontier. Weak scaling efficiencies of 97% and 95% are observed when scaling to 50% of Summit and 95% of Frontier. Strong scaling efficiencies of 84% and 81% are observed when increasing the device count by a factor of 8 and 16 on V100 and MI250X hardware. The strong scaling efficiency of AMD's MI250X increases to 92% when increasing the device count by a factor of 16 when GPU-aware MPI is used for communication.

Figures (10)

• Figure 1: Roofline performance of the most expensive kernels on (a) OLCF Summit and (b) OLCF Frontier. The Riemann solver and WENO kernels use 13% and 45% of peak FLOPS on OLCF Summit. On OLCF Frontier, the Riemann solver and WENO kernels use 3% and 21% of peak FLOPS.
• Figure 2: Weak scaling results on (a) OLCF Summit and (b) OLCF Frontier. On OLCF Summit, MFC scales from 128 V100 GPUs to 13824 V100 GPUs (50% of the machine) with 97% efficiency. On OLCF Frontier, MFC scales from 128 MI250X GCDs to 65536 MI250X GCDs (87% of the machine) with 95% efficiency.
• Figure 3: Strong scaling results on (a) OLCF Summit and (b) OLCF Frontier. On OLCF Summit, a simulation with 8 million cells per GPU maintains 84% of ideal performance when increasing the GPU count by a factor of 8. On OLCF Frontier, a simulation with 32 million cells per GCD maintains 81% of ideal performance when increasing the GCD count by a factor of 16.
• Figure 4: Strong scaling on OLCF Frontier with and without GPU-aware MPI. With GPU-aware MPI, a simulation with 32 million cells per GCD maintains 92% of ideal performance when increasing the GCD count by a factor of 16. This is a 14% increase over the 81% of ideal performance maintained when increasing device count by a factor of 16 without GPU-aware MPI.
• Figure 5: Relative speed up in grind time for the optimized code executed on GPUs, as labeled, compared to the fastest benchmarked CPUs (all cores used) offered by AMD, NVIDIA, Intel, and IBM. Larger numbers indicate a large speedup of the single GPU over the single CPU socket.