GRaM-X: A new GPU-accelerated dynamical spacetime GRMHD code for Exascale computing with the Einstein Toolkit

TL;DR

A new GPU-accelerated dynamical-spacetime general relativistic magnetohydrodynamics (GRMHD) code which extends the GRMHD capability of Einstein Toolkit to GPU-based exascale systems and finds excellent agreement with analytic results and results of other codes reported in literature.

Abstract

We present GRaM-X (General Relativistic accelerated Magnetohydrodynamics on AMReX), a new GPU-accelerated dynamical-spacetime general relativistic magnetohydrodynamics (GRMHD) code which extends the GRMHD capability of Einstein Toolkit to GPU-based exascale systems. GRaM-X supports 3D adaptive mesh refinement (AMR) on GPUs via a new AMR driver for the Einstein Toolkit called CarpetX which in turn leverages AMReX, an AMR library developed for use by the United States DOE's Exascale Computing Project (ECP). We use the Z4c formalism to evolve the equations of GR and the Valencia formulation to evolve the equations of GRMHD. GRaM-X supports both analytic as well as tabulated equations of state. We implement TVD and WENO reconstruction methods as well as the HLLE Riemann solver. We test the accuracy of the code using a range of tests on static spacetime, e.g. 1D MHD shocktubes, the 2D magnetic rotor and a cylindrical explosion, as well as on dynamical spacetimes, i.e. the oscillations of a 3D TOV star. We find excellent agreement with analytic results and results of other codes reported in literature. We also perform scaling tests and find that GRaM-X shows a weak scaling efficiency of $\sim 40-50\%$ on 2304 nodes (13824 NVIDIA V100 GPUs) with respect to single-node performance on OLCF's supercomputer Summit.

Figures (8)

• Figure 1: A sample of data arrangements supported by CarpetX, displaying different types of variables. From left to right: fully vertex centered, face centered in the $y$ direction, and fully cell centered. Notice that there is an extra value in any vertex centered direction compared to cell centered directions. This figure was modeled after amrex-docs:web.
• Figure 2: Results for the evolution of 1D MHD shocktube tests Balsara 1 and Balsara 4 in the left and the right panel respectively. We plot the density $\rho$, pressure $P$, normal velocity $v^x$, tangential velocity $v^y$, Lorentz factor $W$ and tangential component of the magnetic field $B^y$ at time $t_{\rm{ref}}=0.4$ for both tests. We choose the initial conditions as listed in Table \ref{['tab:balsara']} on a single-level grid between $[-0.5,0.5]$ with a spatial resolution $\Delta x = 1/1600$ and evolve with a time resolution $\Delta t = 5 \times 10^{-4}$. We find that our results agree well with the analytical solution giacomazzo_rezzolla_2006 (limited by numerical errors) and any minor deviations are consistent with those reported by other codes Balsara_2001Giacomazzo_2007Moesta_GRHydro_2013. This test took 29 seconds to evolve on an NVIDIA RTX A6000 GPU.
• Figure 3: Same as Fig. \ref{['fig:balsara14']} but for Balsara 2, Balsara 3 and Balsara 5. We plot the results at time $t_{\rm{ref}}=0.4$ for Balsara 2 and 3, and at time $t_{\rm{ref}}=0.55$ for Balsara 5. We find excellent agreement with the analytical results giacomazzo_rezzolla_2006 for Balsara 2 and Balsara 5. Balsara 3 is the most demanding of all Balsara tests and we find that there is a considerable undershoot near the right moving fast shock for $\rho$, $v^y$ and $B^y$. However, using WENO reconstruction instead of TVD reconstruction at the same resolution resolves this issue to a large extent and makes the deviations comparable to those observed by Balsara_2001Moesta_GRHydro_2013
• Figure 4: Left panel: The evolved state of the 2D cylindrical explosion test at time $t = 4$. We plot the logarithm of pressure, the Lorentz factor as well as the $x$ and $y$ components of the magnetic field. We also overlay the magnetic field lines on top of the Lorentz factor plot. We perform the test on a single level grid between $-6 < x, y < 6$ with a spatial resolution of $\Delta x = \Delta y = 0.06$ and evolve with a temporal resolution of $\Delta t = 0.015$. We find that our results agree well with the results reported by Moesta_GRHydro_2013 and Komissarov:1999 for the same test setup. Right panel: 1D profile along $x=0$ and $y=0$, for density $\rho$, pressure $P$, magnetic pressure $b^2/2$ and Lorentz factor $W$. We find overall good agreement but minor deviations with Fig. 6 of Moesta_GRHydro_2013 for density and magnetic pressure. These deviations occur because Moesta_GRHydro_2013 uses vertex-centering while GRaM-X uses cell-centering for hydrodynamic variables. The deviations go away when we perform the test with same coordinate discretization as Moesta_GRHydro_2013. This test took 15 seconds to evolve on an NVIDIA RTX A6000 GPU.
• Figure 5: Left panel: The evolved state of the 2D magnetic rotor test at time $t = 0.4$. We plot the density, pressure, magnetic pressure and Lorentz factor, and also overlay the magnetic field lines on top of the Lorentz factor plot. We perform the test on a single-level grid between $-0.5 < x, y < 0.5$ with a spatial resolution of $\Delta x = \Delta y = 1/400$ and evolve with a temporal resolution of $\Delta t = 6.25 \times 10^{-4}$. These 2D results show very good agreement with Zanna:2003, Etienne:2010 and Moesta_GRHydro_2013. Right panel: 1D profiles along $x=0$ and $y=0$ for density $\rho$, pressure $P$, magnetic pressure $b^2/2$ and Lorentz factor $W$. Again, we find excellent agreement overall but some minor deviations with Fig. 6 of Moesta_GRHydro_2013 and Fig. 9 of Etienne:2010 due to different centering of variables. In fact, these deviations becomes irrelevant when we compare the results of other codes at different resolutions (see Fig. 9 of Etienne:2010 for example). This test took 97 seconds to evolve on an NVIDIA RTX A6000 GPU.