Framework and Methodology for Verification of a Complex Scientific Simulation Software, Flash-X

TL;DR

The paper addresses the challenge of verifying a highly composable, multiphysics simulation framework, Flash-X, where correct functionality must be maintained across numerous component permutations. It introduces a scaffolded testing methodology and a FlashTest-driven framework that constructs platform-specific test.info files from repository-defined tests (tests.yaml) and suite definitions, enabling automated, reproducible verification across diverse environments. Key contributions include the concept of test scaffolding to achieve broad coverage without exhaustive permutations, a workflow for initialization, suite setup, and execution via the flashxtest toolkit, and a governance model with gatekeepers to balance development needs with quality assurance. The approach provides practical guidance for verifying complex scientific software and can serve as a template for similar open-source, gatekept projects dealing with highly configurable, PDE/ODE-based simulations and AMR-enabled solvers.

Abstract

Computational science relies on scientific software as its primary instrument for scientific discovery. Therefore, similar to the use of other types of scientific instruments, correct software and the correct operation of the software is necessary for executing rigorous scientific investigations. Scientific software verification can be especially difficult, as users typically need to modify the software as part of a scientific study. Systematic methodologies for building test suites for scientific software are rare in the literature. Here, we describe a methodology that we have developed for Flash-X, a community simulation software for multiple scientific domains, that has composable components that can be permuted and combined in a multitude of ways to generate a wide range of applications. Ensuring sufficient code coverage by a test suite is particularly challenging due to this composability. Our methodology includes a consideration of trade-offs between meeting software quality goals, developer productivity, and meeting the scientific goals of the Flash-X user community.

Figures (8)

• Figure 1: Providing unit-test-like coverage through scaffolding for code sections that are not easily amenable to unit testing. Code components, A, B, C and D can be tested using different combinations of tests, A1, B1, C1-C3 and D1-D4 based on their mutual inter-dependence. The code snippet covers conditions and possible combinations of the scaffolding.
• Figure 2: Types of regression test requirements for simulation software instruments.
• Figure 3: Schematic of Yaml and Suite files that are combined to produce a test.info file (see Section \ref{['testframework']}) to run tests. The corresponding test.info is shown in Fig. \ref{['fig:flashtest']}, and the details related to the structure of the files are provided in Section \ref{['sc:tooling']}.
• Figure 4: Example of configuration files for FlashTest. The config file is created through the command-line invocation and test.info is generated by combining the tests.yaml and suite files, as shown in Fig. \ref{['fig:suite-setup']}. Details are provided in Section \ref{['sc:tooling']}.
• Figure 5: Errors can be detected in the output files (1-4) in a simulation with FlashTest.