The auto_tmpdir Plugin for Per-job Temporary Directories

Many jobs that run on a cluster create temporary files of some kind:

• Open MPI stores information regarding the topology and configuration of the worker processes in $TMPDIR
• VALET stores environment snapshots (on each vpkg_require) in $TMPDIR
• The PSM2 library (for accelerated MPI over Intel OPA) and VADER plugin create shared memory segments in /dev/shm
• Some programs are hard-coded to use the /tmp directory directly (not $TMPDIR)

It is helpful to have Slurm automatically manage the lifetime of temporary files associated with jobs running on the cluster.

The Linux virtual file system (VFS)- is a hierarchy of directories and files, referenced by path. That namespace is a single tree, unlike Windows where each distinct storage component is assigned a letter (your primary hard drive is your "C" drive, etc.). In Linux, though, the directories and files on each storage component are mounted at some position within the single VFS tree. On DARWIN, IT-maintained software can be found under /opt/shared on every node, yet those directories and files are not stored in disks present in that node; rather, they are stored on a file server whose storage is mounted at /opt/shared on the nodes. Interacting with the /opt directory requires Linux to access a local disk in the node, but moving to /opt/shared requires interaction with a file server across the network, all transparent to the user who is accessing the file system.

A bind mount in Linux takes an existing directory in the VFS tree and mounts it at another path. This seems like a solution to our problem above: if the per-job temporary directory (e.g. /tmp/job-8451) were bind-mounted at /tmp, then programs that use /tmp rather than $TMPDIR would not leave files in /tmp when they crash — the files would be in /tmp/job-8451. By the same token, bind-mounting /dev/shm/job-8451 at /dev/shm will contain the shared memory segments created by PSM2 and VADER in Open MPI jobs.

Unfortunately, once /tmp/job-8451 is bind-mounted as /tmp every program on the node (including Slurm itself) will store its temporary files in /tmp/job-8451. If the node is shared by multiple jobs this would be a major problem, since each subsequent job that starts will modify what's mounted at /tmp:

Once job 8456 has altered what's mounted at /tmp, job 8451 will no longer see the temporary files it created in /tmp/job-8451 and the program will likely crash. The same problem would exist for Slurm and OS programs that were using files in /tmp prior to job 8451's executing.

For the bind-mount solution to work, each Slurm job needs to have its own VFS tree that is independent of other programs on the node. Linux mount namespaces are exactly that:

• every program that executes starts with its parent's VFS tree
• if the program has appropriate privileges, it can clone that initial VFS tree
• storage components subsequently mounted/unmounted by the program only affect its own VFS tree

For Slurm jobs this equates to:

1. When the job starts, the plugin creates a per-job temporary directory (/tmp/job-8451)
2. The plugin clones the VFS tree (now has a private mount namespace)
3. The plugin bind-mounts /tmp/job-8451 as /tmp (/tmp/job-8451 no longer visible to this program)
4. When the job ends, the plugin unmounts /tmp (/tmp/job-8451 is again visible to this program)
5. The plugin removes /tmp/job-8451

With the same procedure applied to /dev/shm, the major sources of orphaned temporary files would be contained.

The auto_tmpdir Slurm plugin creates the following paths and bind-mounts them:

In some cases the user may want the /tmp directory for the job to be shared by all nodes participating on the job — e.g. somewhere on /lustre. The auto_tmpdir plugin implements a –use-shared-tmpdir flag to the salloc/srun/sbatch commands to request this:

A variant on the shared temporary directory scheme is to have each node use its own separate subdirectory (–use-shared-tmpdir=per-node):

When using the –use-shared-tmpdir flag, the plugin can also be asked to not remove the directories when the job exits by including the –no-rm-tmpdir flag.

The source code for the auto_tmpdir plugin is publicly available on Github.