In order to schedule any job (interactively or batch) on a cluster, you must set your allocation workgroup to define your allocation group and explicitly specify a single partition that corresponds to the compute resources granted for the allocation. For example,

[traine@login00 ~]$ workgroup -g it_css
[(it_css:traine)@login00 ~]$

will set the allocation workgroup to it_css for account traine which is reflected in the prompt change [(it_css:traine)@login00 ~]$ showing the allocation workgroup.

When you submit a job, you must explicitly request a single partition that corresponds to the compute resources granted for the allocation. Keep in mind job scheduling is very complex. It doesn't get considered for execution immediately upon submission. Slurm will analyze and determine on each scheduling cycle, only the next N jobs that are pending will be considered for execution. This means the more jobs submitted by users will likely mean the longer your job may have to wait to be considered. To this point, all users should be good citizens and not over submit, and be patient and do not kill jobs and resubmit to try to increase your priority.

Need help? See Introduction to Slurm in UD's HPC community cluster environment.

All interactive jobs should be scheduled to run on the compute nodes, not the login/head node.

An interactive session (job) can often be made non-interactive (batch job) by putting the input in a file, using the redirection symbols < and >, and making the entire command a line in a job script file:

Then the non-interactive (batch job) job can be scheduled as a batch job.

Remember you must specify your workgroup to define your allocation workgroup for available compute node resources before submitting any job as well as specifying a partition, and this includes starting an interactive session. Now use the Slurm command salloc on the login (head) node. Slurm will look for a node with a free scheduling slot (processor core) and a sufficiently light load, and then assign your session to it. If no such node becomes available, your salloc request will eventually time out.

Type

to start a remote interactive shell on a node in the standard partition use

[(it_css:traine)@login00.darwin ~]$ salloc --partition=standard
salloc: Granted job allocation 39
salloc: Waiting for resource configuration
salloc: Nodes r1n00 are ready for job
[(it_css:traine)@r1n00 ~]$

Type

to open a shell on the login node itself and execute a series of srun commands against that allocation. For example,

[(it_css:traine)@login00.darwin ~]$ salloc --partition=standard --nodes=2 /bin/bash -i
salloc: Granted job allocation 41
salloc: Waiting for resource configuration
salloc: Nodes r1n[00-01] are ready for job
[(it_css:traine)@login00.darwin ~]$

sets up the salloc interactive session with 2 nodes on the standard partition. Now each use of srun is run on all compute nodes inside the interactive session and represents a job step.

Type

    exit

to terminate the interactive shell and release the scheduling slot(s). All the above commands work only when the user is already inside the workgroup and had defined a partition. If you do not specify a workgroup or --partition, you will get an error similar to this

[traine@login00.darwin ~]$ salloc
salloc: error: Please choose a workgroup before submitting a job
salloc: error: cli_filter plugin terminated with error

or

[(it_css:anita)@login00.darwin ~]$ salloc
salloc: error: All jobs must explicitly request a partition
salloc: error: cli_filter plugin terminated with error
[(it_css:anita)@login00.darwin ~]$

There is a no way to avoid running the workgroup command and specifying a partition before submitting a job or requesting an interactive session.

You may use the login (head) node for interactive program development including Fortran, C, and C++ program editing and compiling since you won't be billed for usage on the login node. However you will need to use Slurm (salloc) to start interactive shells to utilize your allocation compute node resources for testing or running applications.

A batch job is a command to be executed now or any time in the future. Batch jobs are encapsulated as a shell script (which will be called a job script). This job script is simply a bash script contains special comment lines that provide flags to Slurm to influence their submission and scheduling. Both the srun and salloc command attempt to execute remote commands immediately; if resources are not available they will not return until resources have become available or the user cancels them (by means of <Ctrl>-C).

Slurm provides the sbatch command for scheduling batch jobs:

For example,

 sbatch myproject.qs

This file myproject.qs will contain bash shell commands and SBATCH statements that include SBATCH options and resource specifications. The SBATCH statements begin with # and are the special directives that tell Slurm options such as partition, number of cores, how much time, how much memory, etc. to use for your job.

In every batch session, Slurm sets environment variables that are useful within job scripts. Here are some common examples. The rest ca be found online in Slurm documentation.

When Slurm assigns one of your job's tasks to a particular node, it creates a temporary work directory on that node's 2 TB local scratch disk. And when the task assigned to that node is finished, Slurm removes the directory and its contents. The form of the directory name is

/tmp

For example, after typing salloc –partition=standard on the head node, an interactive job 46 ($SLURM_JOB_ID) is allocated on node r1n00

[(it_css:traine)@login00.darwin ~]$ salloc --partition=standard
salloc: Granted job allocation 46
salloc: Waiting for resource configuration
salloc: Nodes r1n00 are ready for job
[traine@r00n45 ~]$ echo $TMPDIR
/tmp
[(it_css:traine)@r1n00 ~]$

and now we are ready to use our interactive session on node r1n00 and the temporary node scratch directory for this interactive job.

See Filesystems and Computing environment for more information about the node scratch filesystem and using environment variables.

The table below lists sbatch's common options.

Slurm tries to satisfy all of the resource-management options you specify in a job script or as sbatch command-line options.

If no --time=<time-spec> option is specified, then the default time allocated is 30 minutes.

The <time-spec> can be of the following formats:

• <#> - minutes
• <#>:<#> - minutes and seconds
• <#>:<#>:<#> - hours, minutes, and seconds
• <#>-<#> - days and hours
• <#>-<#>:<#> - days, hours, and minutes
• <#>-<#>:<#>:<#> - days, hours, minutes, and seconds

Thus, specifying --time=4 indicates a wall time limit of four minutes and --time=4-0 indicates four days.

In the above case, the former is interpreted as 1 minute with a trailing second argument of "00:00:00" while the second equals 1 day.

The number of CPU cores associated with a job and the scheme by which they are allocated on nodes can be controlled loosely or strictly by the flags mentioned above. Omitting all such flags implies a default will be set to a single task on a single node meaning 1 CPU core will allocated for your job.

For example, putting the lines

#SBATCH --time=60
#SBATCH --ntasks=4

the job script tells Slurm to set a hard limit of 1 hour on the CPU time resource for the job, and requests 4 tasks to be allocated mapped with single processor to each.

When reserving memory for your job by using --mem or --mem-per-cpu option, it will be considered MB if no units are specified, otherwise use the suffix k|M|G|T denoting kibi,mebi,gibi and tebibyte as the units. By default, if no memory specifications are provided, Slurm will allocate 1G per core for your job. For example, specifying

--mem=8G

tells Slurm to reserve 8 gibibyte units of memory for your job. However, specifying the following two options

--mem-per-cpu=8G --ntasks=4

tells Slurm to allocate 8 gibibyte units of memory per core for a total of 32 gigibyte units of memory for your job.

Specifying the correct node type and amount of memory is important because your allocation is billed based on the Service Unit (SU) and each SU varies with the type of node and memory being used. If you specify a node type with a larger amount of memory then you are charged accordingly even if you don't use it.

The table below provides the usable memory values available for each type of node currently available on the DARWIN.

The Extended Memory is accessible by specifying the partition extended-mem and exclusive options. This allows only one user on the node at a time thereby making all swap space accessible for multiple jobs running on that node at once, sharing the swap; but no other user can be on it during that time.

[(it_css:traine)@login00.darwin ~]$ salloc --partition=standard --mem=512G
salloc: error: Memory specification can not be satisfied
salloc: error: Job submit/allocate failed: Requested node configuration is not available
salloc: Job allocation 51 has been revoked.
[(it_css:traine)@login00.darwin ~]$ salloc --partition=large-mem --mem=512G
salloc: Granted job allocation 52
salloc: Waiting for resource configuration
salloc: Nodes r2l00 are ready for job
[(it_css:treine)@r2l00 ~]$

For GPU nodes, you must also specify one of the GPU resource option flags, otherwise your job will not be permitted to run in the GPU partitions.

If a job is submitted with the --exclusive resource, the allocated nodes cannot be shared with other running jobs.

A job running on a node with --exclusive will block any other jobs from making use of resources on that host. To make sure your program is using all the cores on a node when specifying the exclusive resource, include inside the jobs scripts the --ntasks option i.e., --ntasks=36

Job script example:

#SBATCH nodes=2
# The exclusive flag asks to run this job only on all nodes required to fulfill requested slots
#SBATCH --exclusive
#SBATCH --ntasks=36
 
 
 
...

Also, the exclusive resource works in two different ways in Slurm on DARWIN. One is simply specifying --exclusive and the other way is specifying --exclusive=user when submitting a job. In the first method, the job is scaled up with all the resources available on the node irrespective of the requirement. However, the job will only use the number of CPUs specified by the -ntasks option. In the second method, specifying =user means multiple jobs are allowed at the same time on the same node assigned for exclusive access for the user submitting the jobs.

Jobs that will run in one of the GPU partitions must request GPU resources using ONE of the following flags:

If you do not specify one of these flags, your job will not be permitted to run in the GPU partitions.

After entering into the workgroup, GPU nodes can be requested through an interactive session using salloc or through batch submission using sbatch with an appropriate partition name and one of the above GPU resources flag option above.

[(it_css:traine)@login00.darwin ~]$ salloc --partition=gpu-t4 --gpus=1
salloc: Granted job allocation 55
salloc: Waiting for resource configuration
salloc: Nodes r1t00 are ready for job
[(it_css:traine)@r1t00 ~]$ nvidia-smi
Tue Apr 27 11:56:43 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.32.00    Driver Version: 455.32.00    CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:81:00.0 Off |                    0 |
| N/A   42C    P0    26W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
 
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
[(it_css:traine)@r1t00 ~]$

As discussed, an interactive job allows a user to enter a sequence of commands manually. The following qualify as being interactive jobs:

• A program with a GUI: e.g. creating graphs in Matlab
• A program that requires manual input: e.g. a menu-driven post-processing program
• Any task that is more easily performed manually

As far as the final bullet point goes, suppose a user has a long-running batch job and must later extract results from its output using a single command that will execute for a short time (say five minutes). While the user could go to the effort of creating a batch job, it may be easier to just run the command interactively and visually note its output.

In Slurm, interactive jobs are submitted to the job scheduler first by using the salloc command specifying a partition after being in your workgroup:

[(it_css:traine)@login00.darwin ~]$ salloc --partition=standard
salloc: Granted job allocation 56
salloc: Waiting for resource configuration
salloc: Nodes r1n00 are ready for job
[(it_css:traine)@r1n00 ~]$

Dissecting this text of both, we see that:

1. the job was assigned a numerical job identifier or job id of 56
2. the job is assigned to the standard partition with job resources tracked and billed against the allocation group (workgroup), it_css
3. the job is executing on compute node r1n00
4. no cores, memory or time are specified so the defaults defined for all partitions is applied: 1 core, 1G and 30 minutes
5. the final line is a shell prompt, running on r1n00 and waiting for commands to be typed

One can specify all the options that are applicable to sbatch in the above-mentioned table while running salloc command if appropriate.

Here are more examples to reflect on:

[(it_css:traine)@login00.darwin ~]$ salloc --partition=standard --mem=400G
salloc: Granted job allocation 58
salloc: Waiting for resource configuration
salloc: Nodes r1n00 are ready for job
[(it_css:traine)@r1n00 ~]$ exit
exit
salloc: Relinquishing job allocation 58
salloc: Job allocation 58 has been revoked.
[(it_css:traine)@login00.darwin ~]$

[(it_css:traine)@login00.darwin 1201]$ salloc --partition=large-mem --mem=500G
salloc: Granted job allocation 59
salloc: Waiting for resource configuration
salloc: Nodes r2l00 are ready for job
[(it_css:traine)@r2l00 1201]$ exit
exit
salloc: Relinquishing job allocation 59
salloc: Job allocation 59 has been revoked.
[(it_css:traine)@login00.darwin ~]$

What is not apparent from the text for the two examples:

• the shell prompt on compute node r1n00 and r2l00 represents different node types due to partition specified, and has as its working directory, the directory in which the salloc command was typed for user traine in workgroup it_css; first example in the home directory '~' and second in workgroup user directory most likely /lustre/it_css/users/1201
• memory specified as 400G or 500G is the total amount of memory needed for the job which requires specifying the appropriate partition either standard or large-mem due to maximum memory available on a particular node type
• if resources had not been immediately available to this job, the text would have "hung" at "waiting for interactive job to be scheduled …" and later resumed with the message about its being successfully scheduled

By default, salloc will start a remote interactive shell on a node in the cluster. The alternative use is to open a shell on the login node itself and execute a series of srun commands against that allocation:

[(it_css:traine)@login00.darwin ~]$ salloc --partition=standard /bin/bash -i
salloc: Granted job allocation 60
salloc: Waiting for resource configuration
salloc: Nodes r1n00 are ready for job
[(it_css:traine)@login00.darwin ~]$ srun /bin/hostname                
r1n00
[(it_css:traine)@login00.darwin ~]$

The command can have arguments presented to it:

[(it_css:traine)@login00.darwin ~]$ printf "%s - %s\n" "$(hostname)" "$(date)"
r0login0.localdomain.hpc.udel.edu - Tue Apr 27 12:57:22 EDT 2021
[(it_css:traine)@login00.darwin ~]$ srun /bin/bash -c 'printf "%s - %s\n" "$(hostname)" "$(date)"'
r1n00 - Tue Apr 27 12:57:57 EDT 2021
[(it_css:traine)@login00.darwin ~]$

The srun command accepts the same commonly-used options above discussed for sbatch and salloc.

By default, salloc will start a remote interactive shell on a node in the cluster. The alternative use is to open a shell on the login node itself and execute a series of srun commands against that allocation:

[(it_css:traine)@login00.darwin ~]$ salloc --partition=standard /bin/bash -i
salloc: Granted job allocation 61
salloc: Waiting for resource configuration
salloc: Nodes r1n00 are ready for job
[(it_css:traine)@login00.darwin ~]$ srun /bin/hostname                
r1n00
[(it_css:traine)@login00.darwin ~]$

The command can have arguments presented to it:

[(it_css:traine)@login00.darwin ~]$ printf "%s - %s\n" "$(hostname)" "$(date)"
r0login0.localdomain.hpc.udel.edu - Tue Apr 27 12:57:22 EDT 2021
[(it_css:traine)@login00.darwin ~]$ srun /bin/bash -c 'printf "%s - %s\n" "$(hostname)" "$(date)"'
r1n00 - Tue Apr 27 12:57:57 EDT 2021
[(it_css:traine)@login00.darwin ~]$ exit
exit
salloc: Relinquishing job allocation 61
salloc: Job allocation 60 has been revoked.
[(it_css:traine)@login00.darwin ~]$

Remember typing exit relinquishing the interactive job and other interactive jobs started within it as well.

Now another set of examples specifying two nodes and executing a series of srun commands against that allocation:

[(it_css:traine)@login00.darwin ~]$ salloc --partition=standard --nodes=2 /bin/bash -i
salloc: Granted job allocation 63
salloc: Waiting for resource configuration
salloc: Nodes r1n[00-01] are ready for job
[(it_css:traine)@login00.darwin ~]$ hostname
r0login0.localdomain.hpc.udel.edu
[(it_css:traine)@login00.darwin ~]$ srun hostname
r1n01
r1n00
[(it_css:traine)@login00.darwin ~]$ srun date
Tue Apr 27 13:07:56 EDT 2021
Tue Apr 27 13:07:56 EDT 2021
[(it_css:traine)@login00.darwin ~]$ exit
exit
salloc: Relinquishing job allocation 63
[(it_css:traine)@login00.darwin ~]$

Each use of srun inside the salloc session represents a job step and is applied to both nodes. The first use of srun is job step zero (0), the second job step 1, etc. When referring to a specific job step, the syntax is <job-id>.<job-step>. The Slurm accounting and billing mechanisms retain usage data for each job step as well as an aggregate for the entire job.

In order to dedicate (reserve) an entire node to run your programs only, one might want to use --exclusive option. For more details, read about exclusive access.

It can be confusing if a user has many interactive jobs submitted at one time. Taking a moment to name each interactive job according to its purpose may save the user a lot of effort later:

[(it_css:traine)@login00.darwin ~]$ salloc --job-name=test --partition=standard
salloc: Granted job allocation 67
salloc: Waiting for resource configuration
salloc: Nodes r1n00 are ready for job
[(it_css:traine)@r1n00 ~]$ echo $SLURM_JOB_NAME
test
[(it_css:traine)@r1n00 ~]$ exit
exit
salloc: Relinquishing job allocation 67
[(it_css:traine)@login00.darwin ~]$

The name provided with the --job-name command-line option will be assigned to the interactive session/job that the user started versus the default name interact. See Managing Jobs on the sidebar for general information about commands in Slurm to manage all your jobs on DARWIN.

Please review using VNC for X11 Applications as an alternative to X11 Forwarding.

We can launch GUI applications on the DARWIN using X-forwarding technique. However, there are some pre-requisites required in order to launch GUI applications using X-forwarding.

For Windows OS, Xming is an X11 display server which must be installed and running on Windows (Windows XP and later) and a PuTTY session must configured with X11 before launching GUI applications on DARWIN. For help on configuring a PuTTY session with X11 see X-Windows (X11) and SSH document for Windows desktop use.

For Mac OS, SSH connection has to be started with -Y argument, ssh -Y darwin.hpc.udel.edu and XQuartz an X11 display server much be installed and running.

Once a SSH connection is established using X11 (and an X11 display server is running, Xming or XQuartz), below are the steps to be followed to test the session.

Type

[traine@login01.darwin ~]$ workgroup -g it_css

Type

[(it_css:traine)@login01.darwin ~]$ echo $DISPLAY
localhost:13.0

Check if the current session is being run with X11 using xdpyinfo | grep display and the name of the display should match the output above.

[(it_css:traine)@login01.darwin ~]$ xdpyinfo | grep display
name of display:    localhost:13.0

$ xdpyinfo | grep display
PuTTY X11 proxy: unable to connect to forwarded X server: Network error: Connection refused
xdpyinfo:  unable to open display "localhost:10.0".

Once we confirm the session is properly configured with X11 forwarding, now we are ready to launch a GUI application on the compute node.

Type

[(it_css:traine)@login01.darwin ~]$ salloc --x11 -N1 -n1 --partition=standard
salloc: Granted job allocation 76
salloc: Waiting for resource configuration
salloc: Nodes r1n02 are ready for job
[(it_css:traine)@r1n02 ~]$

This will launch an interactive job on one of the compute nodes in the standard partition, in this case r1n02, with default options of one cpu (core), 1 GB of memory and 30 minutes time.

Now the compute node and environment will be ready to launch any program that has a GUI (Graphical User Interface) and be displayed on your local computer display.

Additionally, the --x11 argument can be augmented in this fashion --x11=[batch|first|last|all] to the following effects:

• --x11=first This is the default, and provides X11 forwarding to the first compute hosts allocated.
• --x11=last This provides X11 forwarding to the last of the compute hosts allocated.
• --x11=all This provides X11 forwarding from all allocated compute hosts, which can be quite resource heavy and is an extremely rare use-case.
• --x11=batch This supports use in a batch job submission, and will provide X11 forwarding to the first node allocated to a batch job.

These options can be used and further tested using the above display OR $DISPLAY commands.

As with sbatch under Slurm, the flags inside comments may be overridden by values on the sbatch command line. The job script must:

• use Unix-style newlines
• have its executable bit set (e.g. using the chmod u+x command)
• have the interpreting shell shebang present on its first line

A collection of job script templates are maintained by the IT-HPC staff in the /opt/shared/templates directory on DARWIN. All templates therein are written for the Bash shell (the default shell on Linux).

Batch jobs are submitted to the job scheduler using the sbatch command:

[(it_css:traine)@login00 it_css]$ sbatch job_script_01.qs 
Submitted batch job 1146

Notice that the job name defaults to being the name of the job script; as discussed in the previous section, a job name can also be explicitly provided

job_script_02.qs

#SBATCH --job-name=testing002
#SBATCH --output=my_job_op%j.txt
 
echo "Hello, world."

It has already been demonstrated that command-line options to the sbatch command can be embedded in a job script. Likewise, the options can be specified on the command line. For example:

[(it_css:traine)@login00 it_css]$ sbatch --output 'output%j.txt' job_script_02.qs  //After entering into workgroup
Submitted batch job 1158

The --output option was provided in the queue script and on the command line itself: Slurm will honor options from the command line in preference to those embedded in the script. Thus, in this case the "output%j.txt" provided on the command line overrode the "my_job_op%h.txt" from the job script.

The sbatch command has many options available, all of which are documented in its man page. A few of the often-used options will be discussed here.

There are several default options that are automatically added to every sbatch by Slurm as well as default resource requirements supplied, however an explanation of each is beyond the scope of this section. Providing an alternate value for any of these arguments – in the job script or on the sbatch command line – overrides the default value.

Since batch jobs can run unattended, the user may want to be notified of status changes for a job: when the job begins executing; when the job finishes; or if the job was killed. Slurm will deliver such notifications (as emails) to a job's owner if the owner requests them using the --mail-user option:

Consult the man page for the sbatch command for a deeper discussion of each of the --mail-type states. Valid state names are NONE, BEGIN, END, FAIL, REQUEUE, ALL, TIME_LIMIT_50, TIME_LIMIT_80, TIME_LIMIT_90, TIME_LIMIT, ARRAY_TASKS. The time limit states with numbers indicate a percentage of the full runtime: so enabling TIME_LIMIT_50 will see an email notification being delivered once 50% of the job's maximum runtime has elapsed.

Generally, there are two possible cases when jobs are killed: (1) preemption and (2) walltime configured within the jobs script has elapsed. Checkpointing can be used to intercept and handle the system signals in each of these cases to write out a restart file, perform the cleanup or backup operations, or any other tasks before the job gets killed. Of course, this depends on whether or not the application or software you are using is checkpoint enabled.

"TERM" is the most common system signal that is triggered in both the above cases. However, there is a working logic behind the preemption of job which works as below.

When a job gets submitted to a workgroup-specific partition and resources are tied up by jobs in the idle partition, the jobs in the idle partition will be preempted to make way. Slurm sends a preemption signal to the job (SIGCONT followed by SIGTERM) then waits for a grace period (5 minutes) before signaling again (SIGCONT followed by SIGTERM) then killing it (SIGKILL). However, if the job is able to simply be re-run as-is, the user can submit with --requeue to indicate that an idle job that was preempted should be rerun on the idle partition (possibly restarting immediately on different nodes, otherwise it will need to wait for resources to become available).

For example, using the logic provided in one of the Slurm job script templates, one can catch these signals during the preemption and handle them by performing the cleanup or backing up the job results operations as follows.

#SBATCH --partition=idle
#SBATCH --job-name="atest"
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --time=00:02:00
#SBATCH -o stdout.%j
#SBATCH -e stderr.%j
#SBATCH --export=ALL
 
#
# [EDIT] Define a Bash function and set this variable to its
#        name if you want to have the function called when the
#        job terminates (time limit reached or job preempted).
#
job_exit_handler() {
  # Copy all our output files back to the original job directory:
  cp * "$SLURM_SUBMIT_DIR"
 
  # Don't call again on EXIT signal, please:
  trap - EXIT
  exit 0
}
export UD_JOB_EXIT_FN=job_exit_handler
 
 
#
# [EDIT] By default, the function defined above is registered
#        to respond to the SIGTERM signal that Slurm sends
#        when jobs reach their runtime limit or are
#        preempted.  You can override with your own signals
#        list of signals using this variable -- as in this
#        example, which registers for both SIGTERM and the
#        EXIT pseudo-signal that Bash calls when the script
#        ends.  In effect, no matter whether the job is
#        terminated or completes, the UD_JOB_EXIT_FN will be
#        called.
#
export UD_JOB_EXIT_FN_SIGNALS="SIGTERM EXIT"
 
#Do your normal work here
UD_EXEC python test.py

To catch signals asynchronously in Bash, you have to run commands in the background and "wait" for them to complete. This is why the templates includes a shell function named UD_EXEC when you set UD_JOB_EXIT_FN to a trap function name.

If you implement the restart logic at the start of the script, then you can avoid the signal stuff entirely by using the --requeue option with sbatch. Using this option tells Slurm when the job is preempted, it will automatically be moved back into the queue to execute again.

Equally as important as executing the job is capturing any output produced by the job. By default, all the output(stdout and stderr) is sent to a single file that output file is named according to the formula

slurm-[job id].out

For the weather-processing example above, the output would be found in

[(it_css:traine)@login00 it_css]$ sbatch process_weather.qs
Submitted batch job 1158
[(it_css:traine)@login00 it_css]$ 
#
#   ... some time goes by ...
#
[(it_css:traine)@login00 it_css]$ ls *.o*
slurm-1158.out

The name of the output file can be overridden using the –output command-line option to sbatch. The argument to this option is the name of the file, possibly containing special characters that will be replaced by the job id, job name, etc. See the sbatch man page for a complete description.

An array job essentially runs the same job by generating a new repeated task many times. Each time, the environment variable SLURM_ARRAY_TASK_ID is set to a unique value and its value provides input to the job submission script.

The %A_%a construct in the output and error file names is used to generate unique output and error files based on the master job ID (%A) and the array-tasks ID (%a). In this fashion, each array-tasks will be able to write to its own output and error file.

Example: #SBATCH –output=arrayJob_%A_%a.out

The general form of the SBATCH option is:

–array= start_value - stop_value : step_size

with a step size of 2. For example, the option would be:

–array=1-7:2

with index values of 1,2,5,19,27:

–array=1,2,5,19,27

If you have a multiple jobs where you want to automatically run other job(s) after the execution of another job, then you can use chaining. When you chain jobs, remember to check the status of the other job to determine if it successfully completed. This will prevent the system from flooding the scheduler with failed jobs. Here is a simple chaining example with three job scripts doThing1.qs, doThing2.qs and doThing3.qs.

The running of a job can be held until a particular job completes. This can be done so as to not to "hog" resources or because the output of one job is needed as input for the second. Job dependencies are used to defer the start of a job until the specified dependencies have been satisfied. They are specified with the --dependency option to sbatch in the format.

The --dependency portion of sbatch man page lists the flags that are to be used to implement chain jobs. "type" in the below format indicates the flags to be used to establish dependency.

sbatch --dependency=<type:job_id[:job_id][,type:job_id[:job_id]]> ...

The following do1.qs script does 3 important things.

• If first sleeps for 30 seconds. This gives us time to start dependent jobs.
• Does an ls of a non existent file. There is a non-zero exit code for this command.
• Runs the "hello world" program phostname

do1.qs

#!/bin/bash
#SBATCH --job-name="atest"
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --time=00:02:00
#SBATCH -o stdout.%j
#SBATCH -e stderr.%j
#SBATCH --export=ALL

#----------------------
cd $SLURM_SUBMIT_DIR
date
srun -n 8 sleep 30
date

ls this_file_does_not_exist

srun -n 8 /opt/utility/phostname -F

The same script can be run multiple times to demonstrate the dependency option. afterok and afterany options are used for this purpose to establish dependency.

[(it_css:traine)@login00 it_css]$ sbatch -p debug do1.qs
Submitted batch job 36805
[(it_css:traine)@login00 it_css]$ sbatch --dependency=afterany:36805 do1.qs 
Submitted batch job 36806
[(it_css:traine)@login00 it_css]$ sbatch --dependency=afterok:36805 do1.qs 
Submitted batch job 36807

Job 36806 will only start after the intial run i.e., 36805 has finished execution irrespective of its exit status. This is implemented using afterany flag in the sbatch command. In the other case, job 36807 will start only after the first run i.e., 36805 finishes successfully (runs to completion with an exit code of zero).

The result of "ls" command will not affect the overall status of the job. So it might not always be sufficient to just use afterok in chaining jobs. The other option is that you can manually check the error status of individual commands within a script: The error status for a command is held in the variable $?. This can be checked and we can then force the script to exit. For example we can add the line

if [ $? -ne 0 ] ; then ; exit 1234 ;fi 

do1.qs

#!/bin/bash
#SBATCH --job-name="atest"
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --time=00:02:00
#SBATCH -o stdout.%j
#SBATCH -e stderr.%j
#SBATCH --export=ALL

#----------------------
cd $SLURM_SUBMIT_DIR
date
srun -n 8 sleep 30
date

ls this_file_does_not_exist
if [ $? -ne 0 ] ; then ; exit 1234 ;fi

srun -n 8 /opt/utility/phostname -F

Now, job 36807 will not run after submission as an initial run i.e., 36805 will now exit with a non-zero status because of the if condition included in the above script.

This is how chain jobs can be implemented using dependency option.

The interactive and batch jobs discussed thus far have all been serial in nature: they exist as a sequence of instructions executed in order on a single CPU core. Many problems solved on a computer can be solved more quickly by breaking the job into pieces that can be solved concurrently. If one worker moves a pile of bricks from point A to point B in 30 minutes, then employing a second worker to carry bricks should see the job completed in just 15 minutes. Adding a third worker should decrease the time to 10 minutes. Job parallelism likewise coordinates between multiple serial workers to finish a computation more quickly than if it had been done by a single worker. Parallelism can take many forms, the two most prevalent being threading and message passing. Popular implementations of threading and message passing are the OpenMP and MPI standards.

Sometimes a more loosely-coupled form of parallelism can be used by a job. Suppose a user has a collection of 100 files, each containing the full text of a novel. The user would like to run a program for each file that counts the number of gerunds occurring in the text. The counting program is a simple serial program, but the task can be completed more quickly by analyzing many files concurrently. This form of parallelism requires no threading or message passing, and in Slurm parlance is called an array job.

Need help? See Introduction to Slurm in UD's HPC community cluster environment.

Programs that use OpenMP or some other form of thread parallelism should use the "threads" parallel environment. This environment logically limits jobs to run on a single node only, which in turn limits the maximum number of workers to be the CPU core count for a node.

For more details, please look at the job script template /opt/shared/templates/slurm/generic/thread.qs.

It is the user's responsibility to setup the MPI environment before running the actual MPI job. The job script template found in /opt/shared/templates/slurm/generic/mpi/mpi.qs will setup your job requiring a generic MPI parallel environment. This parallel environment spans multiple nodes and allocates workers by "filling-up" one node before moving on. Slurm looks for the --ntasks-per-node to restrict the allocations per node as part of the filling-up strategy. If it is not specified, then the default way of filling-up proceeds. When a job starts an MPI "machines" file is automatically manufactured and placed in the job's temporary directory at ${TMPDIR}/machines. This file should be copied to a job's working directory or passed directly to the mpirun/mpiexec command used to execute the MPI program.

Like choosing the parallel environment in Grid Engine, choosing the appropriate number of tasks, threads, and CPUs required for the job is an important step in Slurm. A lot of information has been documented as comments in the template job scripts for your better understanding. In addition, below are few Slurm arguments that hold more weight while running a parallel job.

Understanding or having a clear picture of the differences between these arguments is necessary to freely work with parallel jobs.

When a parallel job executes, the following environment variables will be set by Slurm:

The mechanism by which you can spread your job across nodes is a bit more complex. If your MPI job wants N CPUs and you're willing to have as few as M of them running per node, then the maximum node count is µ=⌈N/M⌉.

    sbatch --nodes=1-<µ> --ntasks=<N> --cpus-per-task=1 ...

Order is significant, so for N=20 and you are willing to run 6 or more per node, then use

    sbatch --nodes=1-4 --ntasks=20 --cpus-per-task=1 ...

Do not rely on the output of scontrol show job or squeue with regard to the node count while the job is pending; it will not be accurate. Only once the job is scheduled will it show the actual value.

For example,

    $ sbatch --nodes=3-40 --ntasks=80 --cpus-per-task=1
    #!/bin/bash

    env

    Submitted batch job 701892

    $ scontrol show job 701892
    JobId=701892 JobName=sbatch
       UserId=frey(1001) GroupId=everyone(900) MCS_label=N/A
       Priority=5961 Nice=0 Account=it_nss QOS=normal
       JobState=PENDING Reason=Resources Dependency=(null)
         :
       NumNodes=9-40 NumCPUs=80 NumTasks=80 CPUs/Task=1 ReqB:S:C:T=0:0:*:*

    ....some time goes by....

    $ scontrol show job 701892
    JobId=701892 JobName=sbatch
       UserId=frey(1001) GroupId=everyone(900) MCS_label=N/A
       Priority=5961 Nice=0 Account=it_nss QOS=normal
       JobState=COMPLETED Reason=None Dependency=(null)
         :
       NumNodes=5 NumCPUs=80 NumTasks=80 CPUs/Task=1 ReqB:S:C:T=0:0:*:*

The scheduler found 5 nodes with 80 free CPUs (1@r00n17, 35@r01n03, 35@r01n12, 8@r01n16, 1@r01n50):

    SLURM_NTASKS=80
    SLURM_TASKS_PER_NODE=1,35(x2),8,1
    SLURM_NODELIST=r00n17,r01n[03,12,16,50]
    SLURMD_NODENAME=r00n17

Detailed information pertaining to individual kinds of parallel jobs are provided by UD IT in a collection of job template scripts on a per-cluster basis under the /opt/shared/templates/slurm/generic directory. For example, on DARWIN this directory looks like:

[(it_css:traine)@login00 generic]$ ls -l
total 31
drwxr-sr-x 5 frey sysadmin    5 Sep 14 09:36 mpi
-rwxr-xr-x 1 frey sysadmin 5016 Oct 29 10:06 serial.qs
-rwxr-xr-x 1 frey sysadmin 5438 Jan 14 11:53 threads.qs

The directory layout is self-explanatory: script templates specific for all MPI jobs can be found in the mpi directory; Open MPI is in the openmpi directory, generic MPI in the generic directory, and MPICH can be found in the mpich directory (all under the mpi directory; a template for serial jobs is serial.qs and threads.qs should be used for OpenMP jobs. These scripts are heavily documented to aid in users' choice of appropriate templates and are updated as we uncover best practices and performance issues. Please copy a script templates for new projects rather than potentially using an older version from a previous project. See DARWIN Slurm Job Script Templates for more details.

Need help? See Introduction to Slurm in UD's HPC community cluster environment.

Hearkening back to the text-processing example cited above, the analysis of each of the 100 files could be performed by submitting 100 separate jobs to Slurm, each modified to work on a different file. Using an array job helps to automate this task: each sub-task of the array job is gets assigned a unique integer identifier. Each sub-task can find its sub-task identifier in the SLURM_ARRAY_TASK_ID environment variable.

Consider the following:

#!/bin/bash
 
#SBATCH --job-name=arrayJob
#SBATCH --output=arrayJob_%A_%a.out
#SBATCH --error=arrayJob_%A_%a.err
#SBATCH --array=1-16
#SBATCH --time=01:00:00
#SBATCH --ntasks=1
 
######################
# Begin work section #
######################
 
# Print this sub-job's task ID
echo "My Task ID is : " $SLURM_ARRAY_TASK_ID
 
# Do some work based on the SLURM_ARRAY_TASK_ID
# For example: 
# ./my_process $SLURM_ARRAY_TASK_ID
# 
# where my_process is your executable.

[(it_css:traine)@login00 it_css]$ sbatch --array=1-14 array_demo.qs
Submitted batch job 1176
[(it_css:traine)@login00 it_css]$ ...time passes...
[(it_css:traine)@login00 it_css]$ ls -1 arrayJob_*
arrayJob_1176_1
arrayJob_1176_2
arrayJob_1176_3
arrayJob_1176_4
[(it_css:traine)@login00 it_css]$ cat arrayJob_1176_3
My Task ID is : 82709.3

Four sub-tasks are executed, numbered from 1 through 4. The starting index must be greater than zero, and the ending index must be greater than or equal to the starting index. The step size going from one index to the next defaults to one, but can be any positive integer greater than zero. A step size is appended to the sub-task range as in 2-20:2 – proceed from 2 up to 20 in steps of 2, e.g. 2, 4, 6, 8, 10, et al.

There are essentially two methods for partitioning input data for array jobs. Both methods make use of the sub-task identifier in locating the input for a particular sub-task.

If 100 novels were in files with names fitting the pattern novel_«sub-task-id».txt then the analysis could be performed with the following job script gerund_array.qs :

#!/bin/bash
 
#SBATCH --job-name=gerunds
#SBATCH --output=gerund_count_%a.out
#SBATCH --time=01:00:00
#SBATCH --ntasks=1
 
######################
# Begin work section #
######################
 
# Count gerunds in the file:
./gerund_count "novel_${SLURM_ARRAY_TASK_ID}.txt"

[(it_css:traine)@login00 novels]$ sbatch --array=1-100 gerund_array.qs
Submitted batch job 1176

When complete, the job will produce 100 files named gerund_count_«sub-task-id» where the sub-task-id collates the results to the input files.

An alternate method of organizing the chaos associated with large array jobs is to partition the data in directories: the sub-task identifier is not applied to the filenames but is used to set the working directory for each sub-task. With this kind of logic, the job scriptgerund_arrays.qs looks like:

#!/bin/bash
 
#SBATCH --job-name=gerunds
#SBATCH --output=gerund_count.out
#SBATCH --time=01:00:00
#SBATCH --ntasks=1
 
######################
# Begin work section #
######################
 
cd ${SLURM_ARRAY_TASK_ID}
../gerund_count novel.txt > gerund_count

[(it_css:traine)@login00 novels]$ sbatch --array=1-100 gerund_array.qs
Submitted batch job 1177

When complete, each directory will have a file named gerund_count containing the output of the gerund_count command.

The partitioning scheme can be as complex as the user desires. If the directories were not named "1" through "100" but instead used the name of the novel contained within, an index file could be created containing the directory names, one per line:

Great_Expectations
Atlas_Shrugged
The_Great_Gatsby
  :

The job submission script gerund_array.qs might then look like:

#!/bin/bash
 
#SBATCH --job-name=gerunds
#SBATCH --output=gerund_count.out
#SBATCH --time=01:00:00
#SBATCH --ntasks=1
 
######################
# Begin work section #
######################
 
NOVEL_FOR_TASK=`sed -n ${SLURM_ARRAY_TASK_ID}p index.txt`
cd $NOVEL_FOR_TASK
../gerund_count novel.txt > gerund_count

[(it_css:traine)@login00 novels]$ sbatch --array=1-100 gerund_array.qs
Submitted batch job 1178

The sed command selects a single line of the index.txt file; for sub-task 1 the first line is selected, sub-task 2 the second line, etc.