Deciphering boundary layer dynamics in high-Rayleigh-number convection using 3360 GPUs and a high-scaling in-situ workflow

TL;DR

This work demonstrates fully resolved 3D Rayleigh-Bénard convection DNS at $Ra=10^{12}$ using NekRS on 3360 GPUs and showcases an in-situ visualization workflow with ASCENT to capture high-frequency boundary-layer dynamics. The authors identify the HPC challenges of extreme-scale simulations, implement a zero-copy GPU-to-GPU coupling between NekRS and ASCENT, and validate the approach on JUWELS Booster, while also exploring portability to other systems. Key contributions include detailed scaling studies, a comprehensive RBC dataset spanning $Ra$ and $\Gamma$, and an end-to-end in-situ pipeline that significantly reduces data movement compared with post-processing. The results provide both physical insight into boundary-layer fluctuations at high $Ra$ and practical proof of exascale-ready HPC workflows for turbulent convection with time-resolved visualization, reducing overhead and enabling deeper scientific interpretation.

Abstract

Turbulent heat and momentum transfer processes due to thermal convection cover many scales and are of great importance for several natural and technical flows. One consequence is that a fully resolved three-dimensional analysis of these turbulent transfers at high Rayleigh numbers, which includes the boundary layers, is possible only using supercomputers. The visualization of these dynamics poses an additional hurdle since the thermal and viscous boundary layers in thermal convection fluctuate strongly. In order to track these fluctuations continuously, data must be tapped at high frequency for visualization, which is difficult to achieve using conventional methods. This paper makes two main contributions in this context. First, it discusses the simulations of turbulent Rayleigh-Bénard convection up to Rayleigh numbers of $Ra=10^{12}$ computed with NekRS on GPUs. The largest simulation was run on 840 nodes with 3360 GPU on the JUWELS Booster supercomputer. Secondly, an in-situ workflow using ASCENT is presented, which was successfully used to visualize the high-frequency turbulent fluctuations.

Figures (15)

• Figure 1: JUWELS Booster node architecture.
• Figure 2: Schematic of Rayleigh-Bénard convection setup. The top plate is held at a uniform temperature $T=T_t$, the bottom plate at $T=T_b>T_t$.
• Figure 3: Kinetic energy dissipation as a function of the distance from the bottom boundary for $N_{BL}=5,10,15$ and $Ra=e9$.
• Figure 4: Accessible configurations for different supercomputers for $N_{BL}=10$ (top) and $N_{BL}=15$ (bottom). All estimates assume $p=7$.
• Figure 5: Aspect ratio-dependence of RMS profiles of temperature