To Repair or Not to Repair: Assessing Fault Resilience in MPI Stencil Applications

TL;DR

This work tackles fault resilience in MPI stencil applications by combining the ULFM extension with the Legio fault-resilience framework to continue execution using only non-failed processes, trading some result accuracy for faster recovery. It identifies limitations of Legio for stencil patterns, then introduces a decoupled, decoupled-topology approach and incomplete-topology strategies to preserve topology information across repairs. The iPiC3D application is used as a concrete use case to implement fault-aware changes (including A* routing for particles) and to evaluate two real plasma scenarios under fault injection, showing that meaningful results can be extracted without full rollback in many cases. The study highlights that while graceful degradation is valuable, it does not universally replace checkpoint/restart; a combined strategy can reduce time to result when faults occur near the end of runs, guiding future adaptive recovery decisions.

Abstract

With the increasing size of HPC computations, faults are becoming more and more relevant in the HPC field. The MPI standard does not define the application behaviour after a fault, leaving the burden of fault management to the user, who usually resorts to checkpoint and restart mechanisms. This trend is especially true in stencil applications, as their regular pattern simplifies the selection of checkpoint locations. However, checkpoint and restart mechanisms introduce non-negligible overhead, disk load, and scalability concerns. In this paper, we show an alternative through fault resilience, enabled by the features provided by the User Level Fault Mitigation extension and shipped within the Legio fault resilience framework. Through fault resilience, we continue executing only the non-failed processes, thus sacrificing result accuracy for faster fault recovery. Our experiments on a specimen stencil application show that, despite the fault impact visible in the result, we produced meaningful values usable for scientific research, proving the possibilities of a fault resilience approach in a stencil scenario.

Figures (7)

• Figure 1: Bridge solution to deal with the absence of a process (in red) in the communication seen from the perspective of the blue process. The process receives the data sent by the neighbour's neighbour.
• Figure 2: Comparison between particle movements in the presence of faults with the original and A* algorithms. The particle (the blue circle) should move from the bottom-left process to the bottom-right (following the blue dashed arrow), with a fault on the center-bottom (the one in red). The original algorithm tries to move the particle on the x-axis (following orange arrows), but it cannot due to the fault. The A* algorithm (green arrows) overcomes the limitation by sending the particle upwards first.
• Figure 3: Comparison between the executions of the Weibel instability experiment in the absence and presence of faults respectively. These plots show the intensity of the Magnetic field along the z-axis at iteration 320 (40 after the fault). The unit of measure of the field is $\frac{m_{i} c \omega_p}{e}$, where $m_i$ is the mass of a proton, $e$ is the elementary charge, $c$ is the speed of light in vacuum, and $\omega_{p}$ is the plasma frequency of the protons. Axis are normalised to a factor of $c/\omega_p$.
• Figure 4: Energy measurements of the Weibel instability experiment. For reference, $n_0$ is the plasma density, $m_i$ is the mass of a proton, $c$ is the speed of light in vacuum, and $\omega_{p}$ is the plasma frequency of the protons.
• Figure 5: Time to complete 10 iteration of the algorithm as a function of the iteration for the execution with faults of the Weibel instability experiment.