Both sides previous revision Previous revision Next revision | Previous revision Next revisionBoth sides next revision |
abstract:farber:runjobs:schedule_jobs [2018-10-09 12:05] – [Batch Jobs (qsub)] anita | abstract:farber:runjobs:schedule_jobs [2019-02-22 11:31] – [Memory] ssunkara |
---|
===== Scheduling Jobs on Farber ===== | ====== Scheduling Jobs on Farber ====== |
| |
In order to schedule any job (interactively or batch) on a cluster, you must set your **[[/abstract/farber/app_dev/compute_env#using-workgroup-and-directories|workgroup]]** to define your cluster group or //investing-entity// compute nodes. | In order to schedule any job (interactively or batch) on a cluster, you must set your **[[/abstract/farber/app_dev/compute_env#using-workgroup-and-directories|workgroup]]** to define your cluster group or //investing-entity// compute nodes. |
| |
==== Interactive jobs (qlogin) ==== | ===== Interactive jobs (qlogin) ===== |
| |
As discussed, an //interactive job// allows a user to enter a sequence of commands manually. The following qualify as being interactive jobs: | As discussed, an //interactive job// allows a user to enter a sequence of commands manually. The following qualify as being interactive jobs: |
</file> | </file> |
| |
===== Submitting a Batch Job ===== | ==== Submitting a Batch Job ==== |
| |
Grid Engine provides the **qsub** command for scheduling batch jobs: | Grid Engine provides the **qsub** command for scheduling batch jobs: |
where the argument to the ''-a'' option is in the form ''YYYYMMDDHHmm'' (year, month, day, hour, minute). | where the argument to the ''-a'' option is in the form ''YYYYMMDDHHmm'' (year, month, day, hour, minute). |
| |
===== Job Output ===== | ==== Job Output ==== |
| |
Equally as important as executing the job is capturing any output produced by the job. As mentioned above, the ''-j y'' option sends all output (stdout and stderr) to a single file. By default, that output file is named according to the formula | Equally as important as executing the job is capturing any output produced by the job. As mentioned above, the ''-j y'' option sends all output (stdout and stderr) to a single file. By default, that output file is named according to the formula |
<note>If the user overrides the default joining of regular and error output to a single file (using ''-y n''), the error output is directed to a file named as described above but with a ''.e[job id]'' suffix. Likewise, an explicit filename can be provided using the ''-e'' option.</note> | <note>If the user overrides the default joining of regular and error output to a single file (using ''-y n''), the error output is directed to a file named as described above but with a ''.e[job id]'' suffix. Likewise, an explicit filename can be provided using the ''-e'' option.</note> |
| |
===== Forgetting the Filename ===== | ==== Forgetting the Filename ==== |
| |
A user may mistakenly omit the script filename from the ''qsub'' command. Surprisingly, ''qsub'' does not complain in such a situation; instead, it pauses and allows the user to type a script: | A user may mistakenly omit the script filename from the ''qsub'' command. Surprisingly, ''qsub'' does not complain in such a situation; instead, it pauses and allows the user to type a script: |
| [$JOB_NAME].pe[$JOB_ID] | Parallel job **error** filename (Usually empty) | | | [$JOB_NAME].pe[$JOB_ID] | Parallel job **error** filename (Usually empty) | |
| |
==== Command options for qsub ==== | ==== More options for qsub ==== |
| |
The most commonly used **qsub** options fall into two categories: //operational// and //resource-management//. The operational options deal with naming the output files, mail notification of the processing steps, sequencing of a series of jobs, and establishing the UNIX environment. The resource-management options deal with the specific system resources you desire or need, such as parallel programming environments, number of processor cores, maximum CPU time, and virtual memory needed. | The most commonly used **qsub** options fall into two categories: //operational// and //resource-management//. The operational options deal with naming the output files, mail notification of the processing steps, sequencing of a series of jobs, and establishing the UNIX environment. The resource-management options deal with the specific system resources you desire or need, such as parallel programming environments, number of processor cores, maximum CPU time, and virtual memory needed. |
For memory you will be concerned about how much is free. Memory resources come as both consumable and sensor driven (not consumable). For example: | For memory you will be concerned about how much is free. Memory resources come as both consumable and sensor driven (not consumable). For example: |
^ memory resource ^ Consumable ^ Explanation ^ | ^ memory resource ^ Consumable ^ Explanation ^ |
| mem_free | No |Memory that must be available BEFORE job can start | | | m_mem_free | Yes |Memory consumed per CPU DURING execution | |
| m_mem_free | Yes |Memory consumed by the job DURING execution | | |
| |
It is usually a good idea to add both resources. The ''mem_free'' complex is sensor driven, and is more reliable for choosing a node for your job. The ''m_mem_free'' is consumable, which means you are reserving the memory for future use. Other jobs, using ''m_mem_free'', may be barred from starting on the node. If you are specifying memory resources for a parallel environment job, the requested memory is multiplied by the slot count. By default, ''m_mem_free'' is defined as 1GB of memory per core (slot), if not specified. | It is usually a good idea to add both resources. The ''mem_free'' complex is sensor driven, and is more reliable for choosing a node for your job. The ''m_mem_free'' is consumable, which means you are reserving the memory for future use. Other jobs, using ''m_mem_free'', may be barred from starting on the node. If you are specifying memory resources for a parallel environment job, the requested memory is multiplied by the slot count. By default, ''m_mem_free'' is defined as 1GB of memory per core (slot), if not specified. |
| |
<code> | <code> |
qsub -l mem_free=20G,m_mem_free=20G -t 1-30 myjob.qs | qsub -l m_mem_free=20G -t 1-30 myjob.qs |
</code> | </code> |
| |
This will submit 30 jobs to the queue, with the SGE_TASK_ID variable set for use in the ''myjobs.qs'' script (an [[general:userguide:06_runtime_environ?&#array-jobs|array job]].) | This will submit 30 jobs to the queue, with the SGE_TASK_ID variable set for use in the ''myjobs.qs'' script (an [[:abstract:farber:runjobs:schedule_jobs#array-jobs|array job]].) |
The ''mem_free'' resource will cause Grid Engine to find a node (or wait until one is available) with 20 Gbytes of memory free. | The ''m_mem_free'' resource will tell Grid Engine to not schedule a job on a node unless the specified amount of memory i.e., 20GB per CPU is available to consume on that node. Since this is a serial job that runs on a single CPU,20GB can be termed as total memory available for the job. |
The ''m_mem_free'' resource will tell Grid Engine to not schedule a job on a node unless the specified amount of memory is available to consume on that node. | |
==== Parallel environments ==== | ==== Parallel environments ==== |
| |
=== The threads parallel environment === | === The threads parallel environment === |
| |
Jobs such as those having openMP directives use the **//threads//** parallel environment, an implementation of the [[:general:userguide:05_development#programming-models|shared-memory programming model]]. These SMP jobs can only use the cores on a **single** node. | Jobs such as those having openMP directives use the **//threads//** parallel environment, an implementation of the [[:abstract:farber:app_dev:prog_env#programming-models|shared-memory programming model]]. These SMP jobs can only use the cores on a **single** node. |
| |
For example, if your group only owns nodes with 24 cores, then your ''–pe threads'' request may only ask for 24 or fewer slots. Use Grid Engine's **qconf** command to determine the names and characteristics of the queues and compute nodes available to your investing-entity group on a cluster. | For example, if your group only owns nodes with 24 cores, then your ''–pe threads'' request may only ask for 24 or fewer slots. Use Grid Engine's **qconf** command to determine the names and characteristics of the queues and compute nodes available to your investing-entity group on a cluster. |
==== Array jobs ==== | ==== Array jobs ==== |
| |
An [[:general:jobsched:grid-engine:35_parallelism#array-jobs|array job]] essentially runs the same job by generating a new repeated task many times. Each time, the environment variable **SGE_TASK_ID** is set to a sequence number by Grid Engine and its value provides input to the job submission script. | An array job essentially runs the same job by generating a new repeated task many times. Each time, the environment variable **SGE_TASK_ID** is set to a sequence number by Grid Engine and its value provides input to the job submission script. |
| |
<note tip> | <note tip> |