CRIU -- Checkpoint Restore in Userspace for computational simulations and scientific applications

TL;DR

The paper tackles the need to pause long-running scientific simulations and resume them on other resources via checkpointing. It evaluates CRIU, a Linux user-space checkpoint/restore tool, in an HTCondor/OSPool setting, testing a spectrum of workloads from simple C programs to LAMMPS (serial) and containerized executions. Key findings show CRIU can checkpoint/restore many scenarios, including multi-threaded and forked processes, open files, and networked sessions, but face major limitations such as lack of GPU checkpointing, MPI parallelism challenges, and container/interface constraints that hinder cross-machine restoration. The results suggest CRIU offers a path to preemption and migration for many scientific workloads, but practical use is limited by container interfaces and GPU support, highlighting areas for future improvement in container integration, GPU support, and cross-resource migration.

Abstract

Creating new materials, discovering new drugs, and simulating systems are essential processes for research and innovation and require substantial computational power. While many applications can be split into many smaller independent tasks, some cannot and may take hours or weeks to run to completion. To better manage those longer-running jobs, it would be desirable to stop them at any arbitrary point in time and later continue their computation on another compute resource; this is usually referred to as checkpointing. While some applications can manage checkpointing programmatically, it would be preferable if the batch scheduling system could do that independently. This paper evaluates the feasibility of using CRIU (Checkpoint Restore in Userspace), an open-source tool for the GNU/Linux environments, emphasizing the OSG's OSPool HTCondor setup. CRIU allows checkpointing the process state into a disk image and can deal with both open files and established network connections seamlessly. Furthermore, it can checkpoint traditional Linux processes and containerized workloads. The functionality seems adequate for many scenarios supported in the OSPool. However, some limitations prevent it from being usable in all circumstances.