The Athena++ Adaptive Mesh Refinement Framework: Design and Magnetohydrodynamic Solvers

TL;DR

The paper presents Athena++, a high-performance, extensible AMR framework for astrophysical fluid dynamics that separates mesh infrastructure from physics solvers via a block-based AMR design and dynamic task-based execution. It details non-relativistic and relativistic MHD solvers built within this framework, including general EOS support, diffusion via STS, curvilinear coordinate discretizations, and constrained transport to maintain divergence-free magnetic fields. A comprehensive suite of tests demonstrates robust accuracy across linear and nonlinear MHD problems and confirms scalable performance, with AMR providing substantial speedups when refined regions are localized. The work highlights the framework’s portability, modularity, and potential for multi-physics and heterogeneous computing, underscored by open-source distribution and rigorous validation practices.

Abstract

The design and implementation of a new framework for adaptive mesh refinement (AMR) calculations is described. It is intended primarily for applications in astrophysical fluid dynamics, but its flexible and modular design enables its use for a wide variety of physics. The framework works with both uniform and nonuniform grids in Cartesian and curvilinear coordinate systems. It adopts a dynamic execution model based on a simple design called a "task list" that improves parallel performance by overlapping communication and computation, simplifies the inclusion of a diverse range of physics, and even enables multiphysics models involving different physics in different regions of the calculation. We describe physics modules implemented in this framework for both non-relativistic and relativistic magnetohydrodynamics (MHD). These modules adopt mature and robust algorithms originally developed for the Athena MHD code and incorporate new extensions: support for curvilinear coordinates, higher-order time integrators, more realistic physics such as a general equation of state, and diffusion terms that can be integrated with super-time-stepping algorithms. The modules show excellent performance and scaling, with well over 80% parallel efficiency on over half a million threads. The source code has been made publicly available.

Figures (22)

• Figure 1: Labeling of MeshBlocks (top) and their organization into a quadtree (bottom) for an example uniform grid calculation in two dimensions.
• Figure 2: Left panel: example of cell-centered data (red dots) that must be communicated to neighboring MeshBlocks in two dimensions. The shaded cells are ghost cells that overlap with active cells in the eight neighbors. Right panel: face-centered magnetic fields (red arrows) and edge-centered EMFs (blue dots) that are communicated in this example.
• Figure 3: Same as figure \ref{['fig:mesh-tree']} with AMR.
• Figure 4: Example of communication of cell-centered data between neighboring MeshBlocks at different refinement levels. The lower left panel shows the configuration of the MeshBlocks in a two-dimensional mesh. The upper panels show data communicated between MeshBlocks A and B (using red symbols), while the upper left and lower right panel show data communicated between MeshBlocks A and D (using blue symbols). See the text for a description of the symbols.
• Figure 5: Same as figure \ref{['fig:amrcomm1']} but for face-centered variables.