The Case for ABI Interoperability in a Fault Tolerant MPI

TL;DR

This work argues for integrating an Application Binary Interface (ABI) into transparent checkpointing to enable true portability across MPI implementations. It presents a three-legged stool where the MPI application, the MPI library, and the checkpointing package can be compiled and chosen independently, demonstrated here with Mukautuva as an ABI wrapper and the MANA checkpointing system across MPICH and Open MPI. The experimental evaluation shows that ABI-enabled checkpointing incurs low runtime overhead on micro-benchmarks and real applications, and supports seamless cross-implementation restart, illustrating practical benefits for portability, resilience, and dynamic resource management in HPC environments. The findings suggest that ABI interoperability can be generalized beyond MANA to other checkpointing packages and pave the way for broader use of ABI standards in MPI, including containerized and vendor-closed ecosystems.

Abstract

There is new momentum behind an interoperable ABI for MPI, which will be a major component of MPI-5. This capability brings true separation of concerns to a running MPI computation. The linking and compilation of an MPI application becomes completely independent of the choice of MPI library. The MPI application is compiled once, and runs everywhere. This ABI allows users to independently choose: the compiler for the MPI application; the MPI runtime library; and, with this work, the transparent checkpointing package. Arbitrary combinations of the above are supported. The result is a "three-legged stool", which supports performance, portability, and resilience for long-running computations. An experimental proof-of-concept is presented, using the MANA checkpointing package and the Mukautuva ABI library for MPI interoperability. The result demonstrates that the combination of an ABI-compliant MPI and transparent checkpointing can bring extra flexibility in portability and dynamic resource management at runtime without compromising performance. For example, an MPI application can execute and checkpoint under one MPI library, and later restart under another MPI library. The work is not specific to the MANA package, since the approach using Mukautuva can be adapted to other transparent checkpointing packages.

Figures (6)

• Figure 1: In this example, MANA is running over MPICH, although other MPI libraries are possible. The libmuk.so library dynamically detects the MPICH library at runtime, and loads libmpic-wrap.so. The horizontal dotted line separates the upper and lower half in the split process approach. At checkpoint time, only the memory of the upper half is saved. At restart time, (i) the memory of the upper half is restored; (ii) a fresh copy of the lower half (including Mukatauva) is launched; and the wrapper functions of libmana.so are bound to the wrapper functions of libmuk.so.
• Figure 2: Median Latency of MANA for OSU Micro-benchmarks for MPI_Alltoall. Results are in two groups: Cray MPICH and Open MPI. In each group, there are different configurations: native, and with both Mukautuva and MANA. Note that times for each group are almost identical, indicating that MANA and Mukautuva add little runtime overhead.
• Figure 3: Median Latency of MANA for OSU Micro-benchmarks for MPI_Bcast. Results are in two groups: Cray MPICH and Open MPI. In each group, there are different configurations: native, and with both Mukautuva and MANA. MPI_Bcast is more efficient than MPI_Alltoall. Therefore, we can observe some small runtime overhead with small message sizes. This runtime overhead is due to the lack of a kernel feature in Discovery's old Linux kernel that is required by MANA to improve runtime efficiency.
• Figure 4: Median Latency of MANA for OSU Micro-benchmarks for MPI_Allreduce. Results are in two groups: Cray MPICH and Open MPI. In each group, there are different configurations: native, and with both Mukautuva and MANA. We can observe the same runtime overhead with small message sizes. In addition, the result of Mukautuva and MANA outperformed the native result in some cases because of the larger standard deviation in testing results
• Figure 5: Runtime performance of real-world MPI applications. Here we show the median of the completion time of both applications. The error bars show the standard deviation.