Checkpoint of HPC jobs

Design

Some CRIU operations need root. The main objective here is to do as little as possible using root account. So, we implement a series of scripts to help the dump and restore of processes in the context of HPC.

criu_dump: creates checkpoint in directory ckp-xxxxxx. In order to work properly we need to add capabilities: setcap cap_sys_admin,cap_sys_ptrace+eip /usr/sbin/criu. We need cap_sys_admin in order to be able to use setns syscall. When a process is restored by CRIU, some namespaces are created.

criu_restore_stage1: do some operations before the real restore process. It is mainly used for checkpointing singulairty containers. It creates the cmdline necessary for criu restore command.

criu_wrapper: performs the restore of the process. It needs to be executed as root.

Installing CRIU scripts on the cluster

We only need to clone the repository on the respective directory

mkdir -p /dcsrsoft/spack/external/ckptslurmjob/

The criu_wrapper script should be executed with sudo. Therefore, the following config should be added to sudoers:

ALL ALL=(ALL) NOPASSWD: /dcsrsoft/spack/external/ckptslurmjob/bin/criu_wrapper

Dependencies

criu and crit packages

Using the scripts

Please define the following variables before launching a job:

SBATCH_SIGNAL=B:USR1@60
SBATCH_OPEN_MODE="append" # use to not lose information in out and err files

Scientific-Computing/checkpoint-tools

About

Resources

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages