GPU Acceleration and Portability of the TRIMEG Code for Gyrokinetic Plasma Simulations using OpenMP
Giorgio Daneri
TL;DR
The paper tackles the high computational demand of gyrokinetic TRIMEG simulations by porting the most expensive kernels to GPUs using OpenMP offloading, targeting both AMD and NVIDIA architectures for portability. It details architectural choices, code restructuring, data-management strategies, and debugging approaches to overcome compiler limitations, while evaluating performance through kernel-level analyses, strong scalability, and profiling on multiple HPC clusters. Validation against Cyclone ITG and TCV-X21 benchmarks demonstrates correct physical behavior and comparable growth rates and mode structures, albeit with some differences linked to compiler and library versions. The work provides a practical blueprint for porting complex Fortran-based PIC codes to heterogeneous GPU platforms, highlighting the trade-offs between ease-of-portability and low-level performance control, and offering insights for future compiler and toolchain improvements.
Abstract
The field of plasma physics heavily relies on simulations to model various phenomena, such as instabilities, turbulence, and nonlinear behaviors that would otherwise be difficult to study from a purely theoretical approach. Simulations are fundamental in accurately setting up experiments, which can be extremely costly and complex. As high-fidelity tools, gyrokinetic simulations play a crucial role in discovering new physics, interpreting experimental results, and improving the design of next-generation devices. However, their high computational costs necessitate the use of acceleration platforms to reduce execution time. This work revolves around the TRIangular MEsh based Gyrokinetic (TRIMEG) code, which performs high-accuracy particle-in-cell plasma simulations in tokamak geometries, leveraging a novel finite element approach. The rise of graphical processing units (GPUs) constitutes an occasion to satisfy such computational needs, by offloading the most expensive portion of the code to the accelerators. The chosen approach features GPU offloading with the OpenMP API, which grants portability of the code to different architectures, namely AMD and NVIDIA. The particle pushing as well as the grid-to-particle operations have been ported to GPU platforms. Compiler limitations had to be overcome, and portions of the code were restructured to be suitable for GPU acceleration. Kernel performance was evaluated by carrying out GPU grid size exploration, as well as scalability studies. In addition, the efficiency of hybrid MPI-OpenMP offloading parallelization was assessed. The speedup of the GPU implementation was calculated by comparing it with the pure CPU version using different rationales. The Ion Temperature Gradient (ITG) mode was simulated using the GPU-accelerated version, and its correctness was verified in terms of the energy growth rate and the two-dimensional mode structures.