This is an old revision of the document!
Caviness Slurm Job Script Templates
On both Mills and Farber, example job scripts were made available under the /opt/templates/gridengine
directory. Each template contained extensive Bash code to sense and setup the job environment. Since the script templates were thus self-contained, they had no external dependencies.
One major problem with this paradigm comes when changes must be made to templates. Since users make copies of the template, they are responsible for merging any ongoing change into their copies. In practice, this simply does not happen: a working job script is reused over and over again without modification.
On Caviness, the job script templates have been structured differently. Most of the environment sense and setup code has been shifted out of the job script templates and into external script fragments that are sourced (executed) by the job script. What remains in the job script templates is the setting of variables that influence those external fragments' execute and the sourcing of them. Now, when IT must change the behavior of job scripts, the external fragments are modified and the change is effected for all job scripts deriving from the templates.
Where Can I Find Them?
IT staff are maintaining the Caviness templates via git. The production copy of the repository is checked-out in /opt/shared/slurm/templates
with a symbolic link present at /opt/templates/slurm
to maintain parity with the Mills and Farber systems.
The external script fragments (mentioned above and discussed in detail below) can be found in the /opt/shared/slurm/templates/libexec
directory.
Hierarchical Modularity
Environment setup tasks have been abstracted into each external script fragment file. Examining the fragment directory:
-rw-r--r-- 1 frey sysadmin 1733 Sep 12 2018 common.sh -rw-r--r-- 1 frey sysadmin 5621 May 6 14:11 gaussian.sh -rw-r--r-- 1 frey sysadmin 1580 Sep 12 2018 generic-mpi.sh -rw-r--r-- 1 frey sysadmin 1432 Sep 14 2018 mpich.sh -rw-r--r-- 1 frey sysadmin 4805 Sep 12 2018 openmpi.sh -rw-r--r-- 1 frey sysadmin 2209 May 6 14:11 openmp.sh
common.sh
The common.sh
script contains the basic job environment setup tasks that any job should perform:
- Setting
SLURM_EXPORT_ENV=ALL
to ensure thatsrun
replicates the job's environment in each job step - Setting signal handlers to react to signals delivered by Slurm on preemption and runtime limit (timeout)
- Adding the
UD_EXEC
function to the environment
UD_EXEC and signal handling
The UD_EXEC
function should be used in scripts that register a preemption/timeout signal handler. By default, if a Bash shell is currently executing a command it will not handle any signals until that command has completed executing. Consider a job script like this:
: cleanup() { echo "Time limit exceeded, scrubbing all junk files now" exit 0 } UD_JOB_EXIT_FN=cleanup . /opt/shared/slurm/templates/libexec/common.sh sleep 500000000
When this job is preempted, the SIGTERM
signal is delivered to the Bash shell. The shell notes this, but waits for the sleep
command to finish executing. Since that sleep will last 15.85 years, the 5 minute grace period given to jobs that are preempted or time out expires before the cleanup
function ever gets called, and the job is killed.
In order to get signals to work asynchronously in the Bash shell, long-running commands must be run in the background. The UD_EXEC
function does just that:
: cleanup() { echo "Time limit exceeded, scrubbing all junk files now" exit 0 } UD_JOB_EXIT_FN=cleanup . /opt/shared/slurm/templates/libexec/common.sh UD_EXEC sleep 500000000
The sleep
command is executed in the background and when the SIGTERM
preemption/timeout signal is delivered, the shell immediately calls the cleanup
function as expected.