abstract:farber:runjobs:queues

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
abstract:farber:runjobs:queues [2018-05-22 10:04] – [Farber Exclusive access] sraskarabstract:farber:runjobs:queues [2018-10-08 16:01] (current) – anita
Line 1: Line 1:
-===== The job queues on Farber =====+====== The job queues on Farber ======
  
 Each investing-entity on a cluster an //owner queue// that exclusively use the investing-entity's compute nodes. (They do not use any nodes belonging to others.) Grid Engine allows those queues to be selected only by members of the investing-entity's group. Each investing-entity on a cluster an //owner queue// that exclusively use the investing-entity's compute nodes. (They do not use any nodes belonging to others.) Grid Engine allows those queues to be selected only by members of the investing-entity's group.
Line 19: Line 19:
 ^   <<//investing_entity//>>''.q''  | The default queue for all jobs.| ^   <<//investing_entity//>>''.q''  | The default queue for all jobs.|
 ^  ''standby.q''  | A special queue that spans all standard nodes, at most 200 slots per user.   Submissions will have a lower priority than jobs submitted to owner-queues, and standby jobs will only be started on lightly-loaded nodes.  These jobs will not be preempted by others' job submissions. Jobs will be terminated with notification after running for 8 hours of elapsed (wall-clock) time.  //Also see the ''standby-4h.q'' entry.//  | ^  ''standby.q''  | A special queue that spans all standard nodes, at most 200 slots per user.   Submissions will have a lower priority than jobs submitted to owner-queues, and standby jobs will only be started on lightly-loaded nodes.  These jobs will not be preempted by others' job submissions. Jobs will be terminated with notification after running for 8 hours of elapsed (wall-clock) time.  //Also see the ''standby-4h.q'' entry.//  |
-^  ::: | You must specify **–l standby=1** as a **qsub** option. You must also use the **-notify** option if your jobs traps the USR2 termination signal. [[general:jobsched:standby |(Details)]] |+^  ::: | You must specify **–l standby=1** as a **qsub** option. You must also use the **-notify** option if your jobs traps the USR2 termination signal.|
 ^  ''standby-4h.q''  | A special queue that spans all standard nodes, at most 800 slots per user.   Submissions will have a lower priority than jobs submitted to owner-queues, and 4hr standby jobs will only be started on lightly-loaded nodes.  These jobs will not be preempted by others' job submissions. Jobs will be terminated with notification after running for 4 hours of elapsed (wall-clock) time. | ^  ''standby-4h.q''  | A special queue that spans all standard nodes, at most 800 slots per user.   Submissions will have a lower priority than jobs submitted to owner-queues, and 4hr standby jobs will only be started on lightly-loaded nodes.  These jobs will not be preempted by others' job submissions. Jobs will be terminated with notification after running for 4 hours of elapsed (wall-clock) time. |
-^  ::: | You must specify **–l standby=1** as a **qsub** option. And, if more than 200 slots are requested, you must also specify a maximum run-time of 4 hours or less via the **-l h_rt=//hh:mm:ss//** option. Finally, use the **-notify** option if your jobs traps the USR2 termination signal. [[general:jobsched:standby |(Details)]] |+^  ::: | You must specify **–l standby=1** as a **qsub** option. And, if more than 200 slots are requested, you must also specify a maximum run-time of 4 hours or less via the **-l h_rt=//hh:mm:ss//** option. Finally, use the **-notify** option if your jobs traps the USR2 termination signal.|
 ^  ''spillover.q''  | A special queue that spans all standard nodes and is used by Grid Engine to map jobs when requested resources are unavailable on standard nodes in owner queues, e.g., other standby or spillover jobs are using owner resources. | ^  ''spillover.q''  | A special queue that spans all standard nodes and is used by Grid Engine to map jobs when requested resources are unavailable on standard nodes in owner queues, e.g., other standby or spillover jobs are using owner resources. |
  
Line 27: Line 27:
 In addition to the required run-time limit, the standby queues differ by how many jobs each user can be concurrently running.  More jobs can be run in the standby queues requiring shorter time limits. In addition to the required run-time limit, the standby queues differ by how many jobs each user can be concurrently running.  More jobs can be run in the standby queues requiring shorter time limits.
 </note> </note>
- 
- 
- 
- 
- 
- 
- 
  
 ===== Farber "standby" queues ===== ===== Farber "standby" queues =====
Line 49: Line 42:
 </code> </code>
  
-====Grid Engine resources governing these queues====+==== Grid Engine resources governing these queues ====
  
 The "standby" queues are assigned based on the number of slots and the maximum (wall-clock) //hard// run-time you specify. Typically each cluster defines a default //hard// run-time limit as well as a maximum number of slots allowed per user for standby jobs. The "standby" queues are assigned based on the number of slots and the maximum (wall-clock) //hard// run-time you specify. Typically each cluster defines a default //hard// run-time limit as well as a maximum number of slots allowed per user for standby jobs.
Line 79: Line 72:
  
 ==== Mapping jobs to nodes ==== ==== Mapping jobs to nodes ====
 +
 Once Grid Engine determines the appropriate standby queue, it maps the job to available, idle, nodes (hosts) to fill all the slots. For openmpi jobs, Grid Engine is configured to use the //fill up// allocation rule, by default.  This will keep the number of nodes down, and thus reduce the amount of inter-node communication. Once Grid Engine determines the appropriate standby queue, it maps the job to available, idle, nodes (hosts) to fill all the slots. For openmpi jobs, Grid Engine is configured to use the //fill up// allocation rule, by default.  This will keep the number of nodes down, and thus reduce the amount of inter-node communication.
  
Line 118: Line 112:
 </note> </note>
  
- +==== Actions at the run-time limit =====
- +
-====== Farber Exclusive access ====== +
- +
-If a job is submitted with the [[:general:jobsched:exclusive|exclusive]] resource, Grid Engine will +
- +
-  * promote any serial jobs to 20-core threaded (-pe threads 20)  +
-  * modify any parallel jobs to round-up the slot count to the nearest multiple of 20 +
-  * ignore any memory resources and make all memory available on all nodes assigned to the job +
- +
-A job running on a node with ''-l exclusive=1'' will block any other jobs from making use of resources on that host. +
- +
-Job script example: +
-<code bash> +
-# +
-# The exclusive flag asks to run this job only on all nodes required to fulfill requested slots +
-#$ -l exclusive=1 +
-# +
-# This job needs an openmpi parallel environment using 32 slots = 2 nodes exclusively. +
-#$ -pe openmpi 32 +
-# +
-# By default the slot count granted by Grid Engine will be +
-# used, one MPI worker per slot.  Set this variable if you +
-# want to use fewer cores than Grid Engine granted you (e.g. +
-# when using exclusive=1): +
-# +
-#WANT_NPROC=0 +
- +
-... +
-</code> +
- +
-<note tip>In the script example, this job would be rounded up to 40 and would be assigned 2 nodes. If you really want your job to run with only 32 slots, uncomment and set ''WANT_NPROC=32''.</note> +
- +
-Grid Engine is configured to "fill up" nodes by allocating as many slots as possible before proceeding to another node to fulfill the total number of requested slots for the job.  Unfortunately, Grid Engine may not do what you expect, evenly distribute and fill up across the total number of nodes needed for your job.  For example, if you submit four Open MPI jobs each requesting 20 slots and there are four free nodes each with 20 cores on the cluster, you would expect each job to be assigned on a single node, but in fact, the first job may land on a single node, the second, third, and fourth may wind up straddling the remaining three nodes. +
- +
-To assure that your job will be the only job running on a node (or all nodes needed to satisfy the slots requested), specify the ''exclusive'' resource in the ''qsub'' or ''qlogin'' command, or in a job script. For example, +
- +
-<code text> +
-qsub -l exclusive=1 ... +
-</code> +
- +
- +
- +
-If a job is submitted with the general resource, Grid Engine will +
- +
-  * promote any serial jobs to 20-core threaded (-pe threads 20)  +
-  * modify any parallel jobs to round-up the slot count to the nearest multiple of 20 +
-  * ignore any memory resources and make all memory available on all nodes assigned to the job +
- +
-A job running on a node with ''-l exclusive=1'' will block any other jobs from making use of resources on that host. +
- +
-Job script example: +
-<code bash> +
-# +
-# The exclusive flag asks to run this job only on all nodes required to fulfill requested slots +
-#$ -l exclusive=1 +
-# +
-# This job needs an openmpi parallel environment using 32 slots = 2 nodes exclusively. +
-#$ -pe openmpi 32 +
-# +
-# By default the slot count granted by Grid Engine will be +
-# used, one MPI worker per slot.  Set this variable if you +
-# want to use fewer cores than Grid Engine granted you (e.g. +
-# when using exclusive=1): +
-# +
-#WANT_NPROC=0 +
- +
-... +
-</code> +
- +
-<note tip>In the script example, this job would be rounded up to 40 and would be assigned 2 nodes. If you really want your job to run with only 32 slots, uncomment and set ''WANT_NPROC=32''.</note> +
- +
- +
- +
-=====Actions at the run-time limit=====+
 When a standby job reaches its maximum run time, Grid Engine kills the job. The process depends on your use of Grid Engine's ''-notify'' flag and how your job handles the resulting ''USR2'' notification signal. When a standby job reaches its maximum run time, Grid Engine kills the job. The process depends on your use of Grid Engine's ''-notify'' flag and how your job handles the resulting ''USR2'' notification signal.
  
Line 244: Line 164:
 User             = traine User             = traine
 Queue            = standby.q@n015 Queue            = standby.q@n015
-Host             = n015.mills.hpc.udel.edu+Host             = n015.farber.hpc.udel.edu
 Start Time       = 06/01/2012 12:38:51 Start Time       = 06/01/2012 12:38:51
 End Time         = 06/01/2012 16:43:53 End Time         = 06/01/2012 16:43:53
Line 259: Line 179:
 ==== What if my program does not catch USR2? ==== ==== What if my program does not catch USR2? ====
  
-<note>The following information comes from a lengthy troubleshooting session with a Mills user.  IT thanks the user for his patience while all the details were hashed out.</note>+<note>The following information comes from a lengthy troubleshooting session with a farber user.  IT thanks the user for his patience while all the details were hashed out.</note>
  
 When a program that does not handle the ''USR2'' signal receives that signal, the default action is to abort.  This will raise a ''CHLD'' signal in the program's parent process.  Typically, a Grid Engine job script (the parent process) will react to ''CHLD'' by itself aborting. When a program that does not handle the ''USR2'' signal receives that signal, the default action is to abort.  This will raise a ''CHLD'' signal in the program's parent process.  Typically, a Grid Engine job script (the parent process) will react to ''CHLD'' by itself aborting.
Line 324: Line 244:
 The first ''trap'' command tells the job shell (and child processes) to ignore the ''USR2'' signal.  Appending the ampersand (&) character to the ''mpirun'' command runs it in the background and the job script continues to execute immediately.  Thus, ''mpirun'' has been started and will ignore ''USR2'' signals.  Now, the job shell is set to execute the ''runtimeLimitReached'' function when it itself receives ''USR2'' -- this //does not// change the behavior of the ''mpirun'' that is executing in the background, though (it will still ignore ''USR2'').  Finally, the ''wait'' command puts the job shell to sleep (no CPU usage) until all child processes have exited (namely, ''mpirun'').  The ''wait'' command is a function built-in to BASH and not an external program, so it does not cause the job shell to block and delay reaction to ''USR2'' like ''mpirun'' did in the previous example. The first ''trap'' command tells the job shell (and child processes) to ignore the ''USR2'' signal.  Appending the ampersand (&) character to the ''mpirun'' command runs it in the background and the job script continues to execute immediately.  Thus, ''mpirun'' has been started and will ignore ''USR2'' signals.  Now, the job shell is set to execute the ''runtimeLimitReached'' function when it itself receives ''USR2'' -- this //does not// change the behavior of the ''mpirun'' that is executing in the background, though (it will still ignore ''USR2'').  Finally, the ''wait'' command puts the job shell to sleep (no CPU usage) until all child processes have exited (namely, ''mpirun'').  The ''wait'' command is a function built-in to BASH and not an external program, so it does not cause the job shell to block and delay reaction to ''USR2'' like ''mpirun'' did in the previous example.
  
 +===== Farber Exclusive access =====
 +
 +If a job is submitted with the ''-l exclusive=1'' resource, Grid Engine will
 +
 +  * promote any serial jobs to 20-core threaded (-pe threads 20) 
 +  * modify any parallel jobs to round-up the slot count to the nearest multiple of 20
 +  * ignore any memory resources and make all memory available on all nodes assigned to the job
 +
 +A job running on a node with ''-l exclusive=1'' will block any other jobs from making use of resources on that host.
 +
 +Job script example:
 +<code bash>
 +#
 +# The exclusive flag asks to run this job only on all nodes required to fulfill requested slots
 +#$ -l exclusive=1
 +#
 +# This job needs an openmpi parallel environment using 32 slots = 2 nodes exclusively.
 +#$ -pe openmpi 32
 +#
 +# By default the slot count granted by Grid Engine will be
 +# used, one MPI worker per slot.  Set this variable if you
 +# want to use fewer cores than Grid Engine granted you (e.g.
 +# when using exclusive=1):
 +#
 +#WANT_NPROC=0
 +
 +...
 +</code>
 +
 +<note tip>In the script example, this job would be rounded up to 40 and would be assigned 2 nodes. If you really want your job to run with only 32 slots, uncomment and set ''WANT_NPROC=32''.</note>
 +
 +Grid Engine is configured to "fill up" nodes by allocating as many slots as possible before proceeding to another node to fulfill the total number of requested slots for the job.  Unfortunately, Grid Engine may not do what you expect, evenly distribute and fill up across the total number of nodes needed for your job.  For example, if you submit four Open MPI jobs each requesting 20 slots and there are four free nodes each with 20 cores on the cluster, you would expect each job to be assigned on a single node, but in fact, the first job may land on a single node, the second, third, and fourth may wind up straddling the remaining three nodes.
 +
 +To assure that your job will be the only job running on a node (or all nodes needed to satisfy the slots requested), specify the ''exclusive'' resource in the ''qsub'' or ''qlogin'' command, or in a job script. For example,
 +
 +<code text>
 +qsub -l exclusive=1 ...
 +</code>
 +
 +
 +If a job is submitted with the general resource, Grid Engine will
 +
 +  * promote any serial jobs to 20-core threaded (-pe threads 20) 
 +  * modify any parallel jobs to round-up the slot count to the nearest multiple of 20
 +  * ignore any memory resources and make all memory available on all nodes assigned to the job
 +
 +A job running on a node with ''-l exclusive=1'' will block any other jobs from making use of resources on that host.
 +
 +Job script example:
 +<code bash>
 +#
 +# The exclusive flag asks to run this job only on all nodes required to fulfill requested slots
 +#$ -l exclusive=1
 +#
 +# This job needs an openmpi parallel environment using 32 slots = 2 nodes exclusively.
 +#$ -pe openmpi 32
 +#
 +# By default the slot count granted by Grid Engine will be
 +# used, one MPI worker per slot.  Set this variable if you
 +# want to use fewer cores than Grid Engine granted you (e.g.
 +# when using exclusive=1):
 +#
 +#WANT_NPROC=0
 +
 +...
 +</code>
 +
 +<note tip>In the script example, this job would be rounded up to 40 and would be assigned 2 nodes. If you really want your job to run with only 32 slots, uncomment and set ''WANT_NPROC=32''.</note>
  • abstract/farber/runjobs/queues.1526997883.txt.gz
  • Last modified: 2018-05-22 10:04
  • by sraskar