Revisions to Slurm Configuration v1.1.5 on Caviness

This document summarizes alterations to the Slurm job scheduler configuration on the Caviness cluster.

At the end of June, 2019, the first addition of resources to the Caviness cluster was purchases. The purchase adds:

• 70 new nodes of varying specification
• 2 new kinds of GPU (V100 and T4)
• Several new stakeholder workgroups

Beyond simply booting the new nodes, the Slurm configuration must be adjusted to account for the new hardware, the new workgroups, and adjustments to the resource limits of existing workgroups that purchased Generation 2 nodes.

Our Slurm auto_tmpdir plugin automatically manages per-job (and per-step) temporary file storage:

1. When a job starts, a temporary directory is created and `TMPDIR` is set accordingly in the job's environment
2. Optionally, when a job step starts a temporary directory within the job directory is created and `TMPDIR` is set accordingly in the step's environment
3. As steps complete their temporary directories are removed
4. When the job completes its temporary directory is removed

The plugin offers options to:

1. prohibit removal of the temporary directories
2. create per-job directories in a directory other than /tmp
3. prohibit the creation of per-step temporary directories (steps inherit the job temporary directory)

Normally the plugin creates directories on local scratch storage (/tmp): the temporary files on one node are not visible to other nodes participating in the job. Placing the TMPDIR on Lustre would mean all nodes participating in the job could see the same temporary files. Users could leverage the existing –tmpdir=<path> flag with their own arbitrary path on /lustre/scratch, but that would spread such files across the file system. Having a flag that selects an IT-specified path would keep all shared TMPDIR storage collocated on the file system. This also opens the possibility in the future of having that directory (and all content under it) make use of an OST pool backed by faster media (SSD, NVMe).

Some users have seen the following message in a job output file and assumed their job failed:

auto_tmpdir: remote: failed stat check of <tmpdir> (uid = <uid#>, st_mode = <perms>, errno = <int-code>)

Slurm was reporting this as an error when, in reality, the message was the result of a race condition whereby the job's temporary directory was removed before a job step had completed and removed its own temporary directory (which is inside the job's temporary directory, e.g. /tmp/job_34523/step_1). While it is logically an error in the context of the job scheduler, it is not an error from the point of view of the job or the user who submitted the job.

The following changes to the Slurm configuration will be necessary.

New Generic RESource types will be added to represent the GPU devices present in Generation 2 nodes.

The directory removal error message has been changed to an internal informational message that users will not see. Additional changes were made to the plugin to address the race condition itself.

An additional option has been added to the plugin to request shared per-job (and per-step) temporary directories on the Lustre file system:

--use-shared-tmpdir     Create temporary directories on shared storage (overridden
                        by --tmpdir).  Use "--use-shared-tmpdir=per-node" to create
                        unique sub-directories for each node allocated to the job
                        (e.g. <base>/job_<jobid>/<nodename>).

The addition of new partitions, nodes, and network topology information to the Slurm configuration should not require a full restart of all daemons.

No downtime is expected to be required.