Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SPUMA: a minimally invasive approach to the GPU porting of OPENFOAM

Simone Bnà, Giuseppe Giaquinto, Ettore Fadiga, Tommaso Zanelli, Francesco Bottau

TL;DR

SPUMA provides a minimally invasive GPU port of OpenFOAM by combining a portable programming abstraction, a memory pool with unified memory, and integration with GPU-accelerated solvers (AmgX/GAMG). Demonstrated on Leonardo and LUMI with the DrivAer benchmark, it delivers competitive strong/weak scalability and substantial energy savings versus CPU runs, while enabling incremental porting with minimal code disruption. The approach preserves OpenFOAM's high-level structure and paves the way for community-driven GPU support without extensive reengineering. Future work will target more solvers (compressible, multiphase, heat transfer) and deeper multigrid optimizations to further boost performance.

Abstract

High Performance Computing (HPC) on hybrid clusters represents a significant opportunity for Computational Fluid Dynamics (CFD), especially when modern accelerators are utilized effectively. However, despite the widespread adoption of GPUs, programmability remains a challenge, particularly in open-source contexts. In this paper, we present SPUMA, a full GPU porting of OPENFOAM targeting NVIDIA and AMD GPUs. The implementation strategy is based on a portable programming model and the adoption of a memory pool manager that leverages the unified memory feature of modern GPUs. This approach is discussed alongside several numerical tests conducted on two pre-exascale clusters in Europe, LUMI and Leonardo, which host AMD MI250X and NVIDIA A100 GPUs, respectively. In the performance analysis section, we present results related to memory usage profiling and kernel wall-time, the impact of the memory pool, and energy consumption obtained by simulating the well-known DrivAer industrial test case. GPU utilization strongly affects strong scalability results, reaching 65% efficiency on both LUMI and Leonardo when approaching a load of 8 million cells per GPU. Weak scalability results, obtained on 20 GPUs with the OpenFOAM native multigrid solver, range from 75% on Leonardo to 85% on LUMI. Notably, efficiency is no lower than 90% when switching to the NVIDIA AmgX linear algebra solver. Our tests also reveal that one A100 GPU on Leonardo is equivalent 200-300 Intel Sapphire Rapids cores, provided the GPUs are sufficiently oversubscribed (more than 10 million of cells per GPU). Finally, energy consumption is reduced by up to 82% compared to analogous simulations executed on CPUs.

SPUMA: a minimally invasive approach to the GPU porting of OPENFOAM

TL;DR

SPUMA provides a minimally invasive GPU port of OpenFOAM by combining a portable programming abstraction, a memory pool with unified memory, and integration with GPU-accelerated solvers (AmgX/GAMG). Demonstrated on Leonardo and LUMI with the DrivAer benchmark, it delivers competitive strong/weak scalability and substantial energy savings versus CPU runs, while enabling incremental porting with minimal code disruption. The approach preserves OpenFOAM's high-level structure and paves the way for community-driven GPU support without extensive reengineering. Future work will target more solvers (compressible, multiphase, heat transfer) and deeper multigrid optimizations to further boost performance.

Abstract

High Performance Computing (HPC) on hybrid clusters represents a significant opportunity for Computational Fluid Dynamics (CFD), especially when modern accelerators are utilized effectively. However, despite the widespread adoption of GPUs, programmability remains a challenge, particularly in open-source contexts. In this paper, we present SPUMA, a full GPU porting of OPENFOAM targeting NVIDIA and AMD GPUs. The implementation strategy is based on a portable programming model and the adoption of a memory pool manager that leverages the unified memory feature of modern GPUs. This approach is discussed alongside several numerical tests conducted on two pre-exascale clusters in Europe, LUMI and Leonardo, which host AMD MI250X and NVIDIA A100 GPUs, respectively. In the performance analysis section, we present results related to memory usage profiling and kernel wall-time, the impact of the memory pool, and energy consumption obtained by simulating the well-known DrivAer industrial test case. GPU utilization strongly affects strong scalability results, reaching 65% efficiency on both LUMI and Leonardo when approaching a load of 8 million cells per GPU. Weak scalability results, obtained on 20 GPUs with the OpenFOAM native multigrid solver, range from 75% on Leonardo to 85% on LUMI. Notably, efficiency is no lower than 90% when switching to the NVIDIA AmgX linear algebra solver. Our tests also reveal that one A100 GPU on Leonardo is equivalent 200-300 Intel Sapphire Rapids cores, provided the GPUs are sufficiently oversubscribed (more than 10 million of cells per GPU). Finally, energy consumption is reduced by up to 82% compared to analogous simulations executed on CPUs.
Paper Structure (17 sections, 1 equation, 7 figures, 17 tables)

This paper contains 17 sections, 1 equation, 7 figures, 17 tables.

Table of Contents

  1. Introduction
  2. The OpenFOAM framework
  3. Design aspects
  4. Programming model
  5. Portable programming abstraction
  6. Memory pool
  7. Unified memory
  8. External solvers: amgx4foam
  9. Performance evaluation
  10. Experimental setup
  11. Validation results
  12. Profiling
  13. Scalability test
  14. Energy consumption
  15. Conclusions and future work
  16. ...and 2 more sections

Figures (7)

  • Figure 1: Fixed-size pool: one contiguous memory block of fixed size is divided in many sub-blocks; allocation and deallocation of a new memory space involves only pointer handling.
  • Figure 2: DrivAer model in the notchback (a) and fastback (b) configuration.
  • Figure 3: DrivAer22M. Comparison of velocity field (top), pressure field (middle) and eddy viscosity (bottom) at T=2100 between SPUMA and OpenFOAM-v2412 with PCG as pressure solver.
  • Figure 4: occDrivAer65M. Drag (a) and lift (b) coefficient comparison
  • Figure 5: Coefficient of equivalence (COE) breakdown for a simpleFoam run using SPUMA on 2 GPUs and OpenFOAM-v2412 on 400 CPU cores. The GAMG solver is used for the pressure equation, with the PCG solver and DIC preconditioner for the coarsest level. The simulation is performed on the Leonardo HPC cluster using the DrivAer22M test case.
  • ...and 2 more figures