Large-Scale Simulations of Fully Resolved Complex Moving Geometries with Partially Saturated Cells

TL;DR

This work tackles scalable CFD around complex moving geometries by introducing the Partially Saturated Cells Method (PSM), an LBM extension that uses a solid volume fraction to couple fluid and solid motion on a uniform grid. It couples PSM with a code-generation pipeline (lbmpy) embedded in waLBerla and develops a geometry-field-based mapping to handle rotation/translation efficiently, including a GPU-accelerated interpolation step. The approach achieves high node-level efficiency (e.g., ~71% CPU and ~86% GPU peak) and strong scaling on large HPC systems, demonstrated on a counter-rotating open rotor (CROR) with up to 4.3 billion grid cells and up to 32,768 CPU cores or 1,024 GPUs. The results indicate that PSM enables accurate, scalable simulations of complex moving geometries, with ongoing work to extend validation to turbulent flows, higher Reynolds numbers, and experimental comparisons.

Abstract

We employ the Partially Saturated Cells Method (PSM) to model the interaction between the fluid flow and solid moving objects as an extension to the conventional lattice Boltzmann method. We introduce an efficient and accurate method for mapping complex moving geometries onto uniform Cartesian grids suitable for massively parallel processing. A validation of the physical accuracy of the solid-fluid coupling and the proposed mapping of complex geometries ispresented. The implementation is integrated into the code generation pipeline of the waLBerla framework so that highly optimized kernels for CPU and GPU architectures become available. We study the node-level performance of the automatically generated solver routines. 71% of the peak performance can be achieved on CPU nodes and 86% on GPU accelerated nodes. Only a moderate overhead is observed for the processing of the solid-fluid coupling when compared to the fluids simulations without moving objects. Finally, a counter-rotating rotor is presented as a prototype industrial scenario, resulting in a mesh size involving up to 4.3 billion fluid grid cells. For this scenario, excellent parallel efficiency is reported in a strong scaling study on up to 32,768 CPU cores on the LUMI-C supercomputer and on up to 1,024 NVIDIA A100 GPUs on the JUWELS Booster system.

Figures (13)

• Figure 1: Visualization of the flow around a counter rotating open rotor by the Q-criterion of the flow. (Video available online)
• Figure 2: The fraction field for the Partially Saturated Cells Method representing an example geometry of an ellipse.
• Figure 3: Complete workflow of the code generation pipeline of lbmpy. The code generation for the PSM is integrated in the collision model of the model creation stage. (Figure from suffaArchitectureSpecificGeneration2024).
• Figure 4: Interplay of the fraction field and the super-sampled geometry field for an oval shape with super-sampling depth of 1.
• Figure 5: Counter rotating open rotor geometry with 610,692 vertices and 1,221,380 faces. The front rotor rotating clockwise, while the back rotor (stator) rotating counter-clockwise.