The job queues on Farber

Each investing-entity on a cluster an owner queue that exclusively use the investing-entity's compute nodes. (They do not use any nodes belonging to others.) Grid Engine allows those queues to be selected only by members of the investing-entity's group.

There are also node-wise queues, standby, standby-4h and spillover. Grid Engine allows users to use standard nodes belonging to other investing-entities.

When submitting a batch job to Grid Engine, you specify the resources you need or want for your job. You don't typically specify the name of the queue. Instead, you include a set of directives that specify your job's characteristics. Grid Engine then chooses the most appropriate queue that meets those needs.

The queue to which a job is assigned depends primarily on six factors:

• Whether the job is serial or parallel
• Which parallel environment (e.g., mpi, threads) is needed
• Which or how much of a resource is needed (e.g., max clock time, memory requirements)
• Resources your job will consume (e.g. an entire node, max memory usage)
• Whether the job is non-interactive or interactive

For each investing-entity, the owner-queue names start with the investing-entity's name:

Farber has two standby queues, standby.q and standby-4h.q, that span all "standard" nodes on the cluster. A "standard" node has 20 cores and 64GB of memory.

The "standby" queues span nodes of the cluster to allow you to use unused compute cycles on other groups' nodes in addition to those from your own group's nodes. All standby jobs have a run-time limit specific to a cluster in order to be fair to node owners.

Grid Engine preferentially allocates standby slots on nodes that are lightly loaded. It assigns these jobs a lower queue-priority than jobs submitted by members of the group owning the node. Consequently, normal jobs will tend to push ahead of standby jobs and generally execute before them. Once your job is running, it will not be kill or suspended before the specified run-time limit has been reached.

Specify the standby resource in the qsub command or in a job script to target the standby queues. For example,

qsub -l standby=1 ...

The "standby" queues are assigned based on the number of slots and the maximum (wall-clock) hard run-time you specify. Typically each cluster defines a default hard run-time limit as well as a maximum number of slots allowed per user for standby jobs.

The differences between the two queues is tied to the number of slots and the maximum (wall-clock) hard run-time you specify.

• If you specify a maximum run-time of 4 hours or less (e.g.,h_rt=4:00:00), your job will be assigned to standby-4h.q.

• If you do not specify a maximum run-time or if you specify a run-time greater than 4 hours but not exceeding 8 hours, then you may request up to 200 slots for any job. The job will be assigned to standby.q.

The total number of concurrent slots in the standby.q jobs may not exceed 200, and the total in both standby.q and standby-4h.q may not exceed 800. The limit is by user, and all other jobs will wait for your jobs to finish.

For example, you could concurrently run 25 20-slot jobs (500 slots). This would leave 300 slots available for any other concurrent standby jobs you may submit.

Job script example:

#
# The standby flag asks to run the job in a standby queue.
#$ -l standby=1
#
# This job needs an openmpi parallel environment using 500 slots.
#$ -pe openmpi 500
#
# The h_rt flag specifies a 4-hr maximum (hard) run-time limit.
# The flag is required because the job needs more than 240 slots.
#$ -l h_rt=4:00:00
...

Once Grid Engine determines the appropriate standby queue, it maps the job to available, idle, nodes (hosts) to fill all the slots. For openmpi jobs, Grid Engine is configured to use the fill up allocation rule, by default. This will keep the number of nodes down, and thus reduce the amount of inter-node communication.

It may be useful to control the number of nodes and the number of processes per node. For example:

 qsub -l standby,h_rt=4:00:00 myopenmpi.qs

The MPI processes (ranks) will be mapped to ceiling(NSLOTS/20) hosts. All hosts, except possibly the last host, will have exactly 20 processes. The MPI_FLAG –display_map will display details of the allocations in your Grid Engine output. The MPI_FLAG –loadbalance will spread the processes out among the assigned hosts, without changing the number of hosts.

 #$ -l exclusive=1

 $ qconf -sp mpi | grep allocation_rule
 allocation_rule    $fill_up

 $ qconf -shgrpl
 @128G
 @256G
 ... 

 $ qconf -shgrp @128G
 group_name @128G
 hostlist n068 n069 n070 n071 n106 n108 n109 n110 n113

When a standby job reaches its maximum run time, Grid Engine kills the job. The process depends on your use of Grid Engine's -notify flag and how your job handles the resulting USR2 notification signal.

• Deferred termination: If you include the qsub -notify flag and the job catches the USR2 signal, your job gets an additional 5 minutes of run time during which it can checkpoint itself, log job progress, etc.
• Immediate termination: If your job doesn't include the -notify flag or does not catch the USR2 signal, your job will terminate immediately.

This bash script example catches USR2 and attempts to copy the contents of the job's scratch directory for later use when restarting the job:

#
# The -notify flag to qsub asks that we get a USR2 signal before being killed:
#$ -notify
#
# The -l standby=1 flag asks to run in a standby queue.
#$ -l standby=1
#
#$ -m eas
#$ -M traine@udel.edu
#
 
function copyScratchToLocal {
  echo -n `date`
  echo "  Copying $TMPDIR to "`pwd`
  cp -R $TMPDIR .
}
 
trap copyScratchToLocal SIGUSR2
 
date
ls /etc > $TMPDIR/ls-etc
while [ 1 -eq 1 ]; do
  sleep 10000
done

Submitting this job, the following output is produced:

Fri Jun  1 12:38:51 EDT 2012
User defined signal 2
Fri Jun 1 16:38:53 EDT 2012  Copying /scratch/39866.1.standby.q to /lustre/work/it_css/projects/standby

At two seconds past the max run time (8 hours in this case), the USR2 signal was delivered and the copyScratchToLocal shell function was executed. The job report that Grid Engine emailed when it finally killed this job shows the total run time:

Job 39866.1 (basic.qs) Aborted
Exit Status      = 137
Signal           = KILL
User             = traine
Queue            = standby.q@n015
Host             = n015.farber.hpc.udel.edu
Start Time       = 06/01/2012 12:38:51
End Time         = 06/01/2012 16:43:53
CPU              = 00:00:00
Max vmem         = 17.055M
failed assumedly after job because:
job 39866.1 died through signal KILL (9)

The job ran 5 minutes beyond the 8-hour mark, demonstrating that Grid Engine will indeed wait a full 5 minutes before delivering the KILL signal to end the job.

When a program that does not handle the USR2 signal receives that signal, the default action is to abort. This will raise a CHLD signal in the program's parent process. Typically, a Grid Engine job script (the parent process) will react to CHLD by itself aborting.

Consider an MPI program that does not include the additional code necessary to catch the USR2 signal and gracefully shut itself down (or at least ignore the signal). The job script may look something like this:

  :
function runtimeLimitReached {
  # Do some cleanup, job archiving, whatever, then exit
  # to end the job
 
  exit 0
}
  :
trap runtimeLimitIsReached SIGUSR2
mpirun {arguments to mpirun...}
 
# Do any cleanup that should be done if the job finished prior
# to the runtime limit...

What happens when the runtime limit is reached?

1. The job script and its child mpirun process receive USR2
   • The job script is blocked waiting on the mpirun process, so processing of USR2 is delayed until mpirun has exited
2. Meanwhile, the mpirun process forwards the USR2 signal to all worker processes
3. The worker processes do NOT handle USR2 so they abort upon receiving USR2
4. The mpirun process sees its workers aborting, so it logs some error messages and aborts itself
5. The job script sees the CHLD signal thrown by mpirun and reacts to it by exiting immediately

So the job script never gets a chance to react to the USR2 signal because the CHLD signal coming from mpirun superseded it.

If the job script ignores CHLD then the mpirun would inherit that behavior, ignore all of its children when they abort due to their receiving USR2, and never return control to the job script! This means the job script would still never get a chance to react to USR2.

Without altering the MPI program to handle USR2, there is a way to allow the job script to react to USR2 while the MPI program continues to execute. The job script must be written such that:

1. The job shell has a USR2 signal handler set
2. The mpirun process ignores USR2 (and thus does not propagate it to the worker processes)
3. The mpirun process is executed in such a way as to not block

The sequencing of the commands is the important part:

  :
function runtimeLimitReached {
  # Do some cleanup, job archiving, whatever, then exit
  # to end the job
 
  exit 0
}
  :
trap "" SIGUSR2
mpirun {arguments to mpirun...} &
trap runtimeLimitReached SIGUSR2
wait
 
# Do any cleanup that should be done if the job finished prior
# to the runtime limit...

The first trap command tells the job shell (and child processes) to ignore the USR2 signal. Appending the ampersand (&) character to the mpirun command runs it in the background and the job script continues to execute immediately. Thus, mpirun has been started and will ignore USR2 signals. Now, the job shell is set to execute the runtimeLimitReached function when it itself receives USR2 – this does not change the behavior of the mpirun that is executing in the background, though (it will still ignore USR2). Finally, the wait command puts the job shell to sleep (no CPU usage) until all child processes have exited (namely, mpirun). The wait command is a function built-in to BASH and not an external program, so it does not cause the job shell to block and delay reaction to USR2 like mpirun did in the previous example.

If a job is submitted with the -l exclusive=1 resource, Grid Engine will

• promote any serial jobs to 20-core threaded (-pe threads 20)
• modify any parallel jobs to round-up the slot count to the nearest multiple of 20
• ignore any memory resources and make all memory available on all nodes assigned to the job

A job running on a node with -l exclusive=1 will block any other jobs from making use of resources on that host.

Job script example:

#
# The exclusive flag asks to run this job only on all nodes required to fulfill requested slots
#$ -l exclusive=1
#
# This job needs an openmpi parallel environment using 32 slots = 2 nodes exclusively.
#$ -pe openmpi 32
#
# By default the slot count granted by Grid Engine will be
# used, one MPI worker per slot.  Set this variable if you
# want to use fewer cores than Grid Engine granted you (e.g.
# when using exclusive=1):
#
#WANT_NPROC=0
 
...

Grid Engine is configured to "fill up" nodes by allocating as many slots as possible before proceeding to another node to fulfill the total number of requested slots for the job. Unfortunately, Grid Engine may not do what you expect, evenly distribute and fill up across the total number of nodes needed for your job. For example, if you submit four Open MPI jobs each requesting 20 slots and there are four free nodes each with 20 cores on the cluster, you would expect each job to be assigned on a single node, but in fact, the first job may land on a single node, the second, third, and fourth may wind up straddling the remaining three nodes.

To assure that your job will be the only job running on a node (or all nodes needed to satisfy the slots requested), specify the exclusive resource in the qsub or qlogin command, or in a job script. For example,

qsub -l exclusive=1 ...

If a job is submitted with the general resource, Grid Engine will

• promote any serial jobs to 20-core threaded (-pe threads 20)
• modify any parallel jobs to round-up the slot count to the nearest multiple of 20
• ignore any memory resources and make all memory available on all nodes assigned to the job

A job running on a node with -l exclusive=1 will block any other jobs from making use of resources on that host.

Job script example:

#
# The exclusive flag asks to run this job only on all nodes required to fulfill requested slots
#$ -l exclusive=1
#
# This job needs an openmpi parallel environment using 32 slots = 2 nodes exclusively.
#$ -pe openmpi 32
#
# By default the slot count granted by Grid Engine will be
# used, one MPI worker per slot.  Set this variable if you
# want to use fewer cores than Grid Engine granted you (e.g.
# when using exclusive=1):
#
#WANT_NPROC=0
 
...