FTHP-MPI: Towards Providing Replication-based Fault Tolerance in a Fault-Intolerant Native MPI Library
Sarthak Joshi, Sathish Vadhiyar
TL;DR
The paper tackles the problem of fault tolerance in exascale MPI systems by integrating replication-based resilience with checkpoint/restart within a native MPI library. It introduces FTHP-MPI, a unified framework that wraps a native MPI implementation, enabling coordinated checkpointing and partial replication while preserving high-performance communications. Key contributions include a detailed architectural design for integrating replication with checkpointing, a replica-aware communication mapping, failure management workflows, and extensive experiments showing replication's superiority at large scales with failure rates typical of exascale environments. The work demonstrates practical impact by enabling fault tolerance without modifying application code and by achieving improved efficiency and resilience in failure-prone HPC settings, with overheads remaining negligible under failure-free conditions.
Abstract
Faults in high-performance systems are expected to be very large in the current exascale computing era. To compensate for a higher failure rate, the standard checkpoint/restart technique would need to create checkpoints at a much higher frequency resulting in an excessive amount of overhead which would not be sustainable for many scientific applications. To improve application efficiency in such high failure environments, the mechanism of replication of MPI processes was proposed. Replication allows for fast recovery from failures by simply dropping the failed processes and using their replicas to continue the regular operation of the application. In this paper, we have implemented FTHP-MPI (Fault Tolerance and High Performance MPI), a novel fault-tolerant MPI library that augments checkpoint/restart with replication to provide resilience from failures. The novelty of our work is that it is designed to provide fault tolerance in a native MPI library that does not provide support for fault tolerance. This lets application developers achieve fault tolerance at high failure rates while also using efficient communication protocols in the native MPI libraries that are generally fine-tuned for specific HPC platforms. We have also implemented efficient parallel communication techniques that involve replicas. Our framework deals with the unique challenges of integrating support for checkpointing and partial replication. We conducted experiments emulating the failure rates of exascale computing systems with three applications, HPCG, PIC and CloverLeaf. We show that for large scale systems where the failure intervals are expected to be within a hour, our replication-based library provides higher efficiency and performance than checkpointing-based approaches. We show that under failure-free conditions, the additional overheads due to replication are negligible in our library.