Paper
libyt: an In Situ Interface Connecting Simulations with yt, Python, and Jupyter Workflows
Authors
Shin-Rong Tsai, Hsi-Yu Schive, Matthew J. Turk
Abstract
In the exascale computing era, handling and analyzing massive datasets have become extremely challenging. In situ analysis, which processes data during simulation runtime and bypasses costly intermediate I/O steps, offers a promising solution. We present libyt (https://github.com/yt-project/libyt), an open-source C library that enables astrophysical simulations to analyze and visualize data in parallel computation with yt or other Python packages. libyt can invoke Python routines automatically or provide interactive entry points via a Python prompt or a Jupyter Notebook. It requires minimal intervention in researchers' workflow, allowing users to reuse job submission scripts and Python routines. We describe libyt's architecture for parallel computing in high-performance computing environments, including its bidirectional connection between simulation codes and Python, and its integration into the Jupyter ecosystem. We detail its methods for reading AMR simulations and handling in-memory data with minimal overhead, and procedures for yielding data when requested by Python. We describe how libyt maps simulation data to yt frontends, allowing post-processing scripts to be converted into in situ analysis with just two lines of change. We document libyt's API and demonstrate its integration into two astrophysical simulation codes, GAMER and Enzo, using examples including core-collapse supernovae, isolated dwarf galaxies, fuzzy dark matter, the Sod shock tube test, Kelvin-Helmholtz instability, and the AGORA galaxy simulation. Finally, we discuss libyt's performance, limitations related to data redistribution, extensibility, architecture, and comparisons with traditional post-processing approaches.