Employing Continuous Integration inspired workflows for benchmarking of scientific software -- a use case on numerical cut cell quadrature

TL;DR

The paper addresses benchmarking across competing scientific software packages for numerical integration on geometrically unfitted domains (cut elements). It proposes a CI-driven benchmarking framework built around GitLab pipelines and a MATLAB-based unified driver interface to orchestrate multiple external libraries as submodules and to run both quick and full benchmark modes. Benchmarks generate outputs such as tables, plots, and MATLAB test reports, with artifacts stored for publication and cross-version comparison to assess accuracy, robustness, runtime, and scalability. The framework emphasizes FAIR-aligned practices, reduced overhead for small research teams, and long-term accessibility of benchmarking workflows, offering a transferable approach for diverse scientific software domains.

Abstract

In the field of scientific computing, one often finds several alternative software packages (with open or closed source code) for solving a specific problem. These packages sometimes even use alternative methodological approaches, e.g., different numerical discretizations. If one decides to use one of these packages, it is often not clear which one is the best choice. To make an informed decision, it is necessary to measure the performance of the alternative software packages for a suitable set of test problems, i.e. to set up a benchmark. However, setting up benchmarks ad-hoc can become overwhelming as the parameter space expands rapidly. Very often, the design of the benchmark is also not fully set at the start of some project. For instance, adding new libraries, adapting metrics, or introducing new benchmark cases during the project can significantly increase complexity and necessitate laborious re-evaluation of previous results. This paper presents a proven approach that utilizes established Continuous Integration tools and practices to achieve high automation of benchmark execution and reporting. Our use case is the numerical integration (quadrature) on arbitrary domains, which are bounded by implicitly or parametrically defined curves or surfaces in 2D or 3D.