Portable, heterogeneous ensemble workflows at scale using libEnsemble

TL;DR

libEnsemble addresses exascale challenges by enabling dynamic ensembles via a generator–simulator–allocator paradigm with automatic, portable resource detection across Frontier, Aurora, and Perlmutter. It couples a manager with workers through a shared history array, supports multiple executable paradigms, and enables cross-site operation via Balsam, Globus Compute, and Reverse SSH, highlighting a flexible, resource-aware workflow environment. The paper demonstrates online Gaussian process surrogate training for Wake-T and WarpX plasma accelerator simulations, achieving improvements in RMSE and variance and exhibiting near-ideal weak scaling to over $10^3$ workers, illustrating practical exascale viability. The work advances portable, scalable, and multi-fidelity surrogate workflows and outlines concrete paths for data streaming, domain-specific tooling, and broader generator interoperability. Overall, the framework provides a practical, scalable foundation for dynamic ensembles in exascale scientific computing and multi-fidelity modeling.

Abstract

libEnsemble is a Python-based toolkit for running dynamic ensembles, developed as part of the DOE Exascale Computing Project. The toolkit utilizes a unique generator--simulator--allocator paradigm, where generators produce input for simulators, simulators evaluate those inputs, and allocators decide whether and when a simulator or generator should be called. The generator steers the ensemble based on simulation results. Generators may, for example, apply methods for numerical optimization, machine learning, or statistical calibration. libEnsemble communicates between a manager and workers. We overview the unique characteristics of libEnsemble as well as current and potential interoperability with other packages in the workflow ecosystem. We highlight libEnsemble's dynamic resource features: libEnsemble can detect system resources, such as available nodes, cores, and GPUs, and assign these in a portable way. These features allow users to specify the number of processors and GPUs required for each simulation; and resources will be automatically assigned on a wide range of systems, including Frontier, Aurora, and Perlmutter. Such ensembles can include multiple simulation types, some using GPUs and others using only CPUs, sharing nodes for maximum efficiency. We also describe the benefits of libEnsemble's generator--simulator coupling, which easily exposes to the user the ability to cancel, and portably kill, running simulations based on models that are updated with intermediate simulation output. We demonstrate libEnsemble's capabilities, scalability, and scientific impact via a Gaussian process surrogate training problem for the longitudinal density profile at the exit of a plasma accelerator stage. The study uses gpCAM for the surrogate model and employs either Wake-T or WarpX simulations, highlighting efficient use of resources that can easily extend to exascale.

Figures (13)

• Figure 1: Interoperability among libEnsemble's allocator, generators, and simulators user functions
• Figure 2: Persistent generator function using a model to decide new simulation inputs.
• Figure 3: Simple simulator function
• Figure 4: Configuring and running an ensemble.
• Figure 5: Use of MPI Executor in calling script (above) and simulation function (below). These scripts will work on various systems including Perlmutter, Frontier, and Aurora.