Multi-Partner Project: Multi-GPU Performance Portability Analysis for CFD Simulations at Scale

TL;DR

This work addresses the problem of performance portability for GPU-accelerated CFD by conducting a full-stack, multi-level analysis of the SOD2D spectral element solver across NVIDIA and AMD platforms within the REFMAP workflow. The authors examine hardware, software, and memory-level factors, employing kernel-splitting, preloading, and prefetching to explore a wide design space and quantify cross-vendor disparities using single- and multi-GPU runs. Key findings show significant vendor-dependent differences in kernel throughput and scalability, with memory-optimization gains that vary by precision and device; extrapolating low-scale optimizations to large-scale runs is unreliable. The work’s significance lies in illustrating the necessity for automated, hardware-aware tuning to generate scalable CFD datasets for surrogate training in urban-airflow applications, enabling efficient cross-platform deployment of REFMAP workloads.

Abstract

As heterogeneous supercomputing architectures leveraging GPUs become increasingly central to high-performance computing (HPC), it is crucial for computational fluid dynamics (CFD) simulations, a de-facto HPC workload, to efficiently utilize such hardware. One of the key challenges of HPC codes is performance portability, i.e. the ability to maintain near-optimal performance across different accelerators. In the context of the \textbf{REFMAP} project, which targets scalable, GPU-enabled multi-fidelity CFD for urban airflow prediction, this paper analyzes the performance portability of SOD2D, a state-of-the-art Spectral Elements simulation framework across AMD and NVIDIA GPU architectures. We first discuss the physical and numerical models underlying SOD2D, highlighting its computational hotspots. Then, we examine its performance and scalability in a multi-level manner, i.e. defining and characterizing an extensive full-stack design space spanning across application, software and hardware infrastructure related parameters. Single-GPU performance characterization across server-grade NVIDIA and AMD GPU architectures and vendor-specific compiler stacks, show the potential as well as the diverse effect of memory access optimizations, i.e. 0.69$\times$ - 3.91$\times$ deviations in acceleration speedup. Performance variability of SOD2D at scale is further examined on the LUMI multi-GPU cluster, where profiling reveals similar throughput variations, highlighting the limits of performance projections and the need for multi-level, informed tuning.

Figures (5)

• Figure 1: Overview of REFMAP's HPC to ML workflow, with emphasis on CFD acceleration.
• Figure 2: High-level algorithmic overview of SOD2D
• Figure 3: Organization of the examined multi-level design space.
• Figure 4: Single-GPU run times across all kernel optimizations for Channel Flow.
• Figure 5: Throughput at scale for Channel Flow.