| Both sides previous revision Previous revision Next revision | Previous revision |
| abstract:caviness:runjobs:schedule_jobs [2024-01-24 15:56] – [GPU nodes] anita | abstract:caviness:runjobs:schedule_jobs [2024-09-25 14:10] (current) – [Array Jobs] anita |
|---|
| where '<N>' should be the number of GPUs and depends on the specific [[abstract:caviness:caviness#compute-nodes|compute node specifications]]. | where '<N>' should be the number of GPUs and depends on the specific [[abstract:caviness:caviness#compute-nodes|compute node specifications]]. |
| |
| <note important>**VERY IMPORTANT:** Keep in mind that not all memory can be reserved for a node due to a small amount required for system use. As a result, the maximum amount of memory that can be specified is based on what Slurm shows as available. For example, the baseline nodes in Caviness show a memory size of 124 GiB versus the 128 GiB of physical memory present in them. This means if you try to specify the full amount of memory (i.e. 128G), then Slurm will try to run the job on a larger memory node as long as you have access to a larger memory node. This will work if you specify the standard partition or if you specify a workgroup partition and your research group purchased a larger memory node, otherwise your job will never run.</note> | <note important>**VERY IMPORTANT:** Keep in mind that not all memory can be reserved for a node due to a small amount required for system use. As a result, the maximum amount of memory that can be specified is based on what Slurm shows as available. For example, the baseline nodes in Caviness show a memory size of 124 GiB versus the 128 GiB of physical memory present in them. This means if you try to specify the full amount of memory (i.e. 128G), then Slurm will try to run the job on a larger memory node as long as you have access to a larger memory node. This will work if you specify the standard partition or if you specify a workgroup partition and your research group purchased a larger memory node, otherwise your job will never run. You may also use ''%%--%%mem=0'' to request all the memory on a node.</note> |
| |
| ==== Exclusive access ==== | ==== Exclusive access ==== |
| ==== GPU nodes ==== | ==== GPU nodes ==== |
| |
| After entering into the workgroup, GPU nodes can be requested through an interactive session using ''salloc'' or through batch submission using ''sbatch''. An appropriate partition name (such as a workgroup for running or ''devel'' if you need to compile on a GPU node) has to be mentioned while running the command as below. | After entering into the workgroup, GPU nodes can be requested through an interactive session using ''salloc'' or through batch submission using ''sbatch''. An appropriate partition name (such as a workgroup for running or ''devel'' if you need to compile on a GPU node) and a GPU resource and type **must** be specified while running the command as below. |
| |
| <code bash> | <code bash> |
| </code> | </code> |
| |
| Also if your workgroup has purchased more than one kind of GPU node and you want to target a specific GPU node type, then you can use ''%%--%%gres=gpu:p100'' or ''%%--%%gres=gpu:v100'' or ''%%--%%gres=gpu:t4''. See [[abstract:caviness:runjobs:job_status#sworkgroup|sworkgroup]] to determine your workgroup resources including GPU node type. In the example below, this particular workgroup has (2) ''gpu:p100'' and (2) ''gpu:v100'' types of GPUs available | Also if your workgroup has purchased more than one kind of GPU node, then you need to choose that specific GPU type to target it, such as ''%%--%%gres=gpu:p100'' or ''%%--%%gres=gpu:v100'' or ''%%--%%gres=gpu:t4'' or ''%%--%%gres=gpu:a100'' to by default get 1 GPU or the form ''%%--%%gres=gpu:<<GPU type>>:<<#>''. See [[abstract:caviness:runjobs:job_status#sworkgroup|sworkgroup]] to determine your workgroup resources including GPU node type. In the example below, this particular workgroup has (2) ''gpu:p100'', (2) ''gpu:v100'' and (2) ''gpu:a100'' types of GPUs available |
| |
| <code bash> | <code bash> |
| [traine@login00 ~]$ sworkgroup -g ececis_research --limits | [traine@login00 ~]$ sworkgroup -g ececis_research --limits |
| Partition Per user Per job Per workgroup | Partition Per user Per job Per workgroup |
| ---------------+--------+-------+------------------------------------------------- | ---------------+--------+-------+----------------------------------------------------------------- |
| devel 2 jobs cpu=4 | devel 3 jobs cpu=4 |
| ececis_research cpu=152,mem=1882G,gres/gpu:p100=2,gres/gpu:v100=2 | ececis_research cpu=248,mem=3075G,gres/gpu:p100=2,gres/gpu:v100=2,gres/gpu:a100=2 |
| reserved | reserved |
| standard cpu=720 cpu=360 | standard cpu=720 cpu=360 |
| </code> | </code> |
| |
| Any user can employ a GPU by running in the ''standard'' partition, however keep in mind jobs can be preempted and would require [[abstract:caviness:runjobs:schedule_jobs#handling-system-signals-aka-checkpointing|checkpointing]] as part of your batch job script. The interactive session example below requests any node with a GPU v100, 1 core and 1 GB of memory (default values if not specified) on the standard partition. | Any user can employ a GPU by running in the ''standard'' partition, however keep in mind a GPU type **must** be specified, jobs can be preempted and would require [[abstract:caviness:runjobs:schedule_jobs#handling-system-signals-aka-checkpointing|checkpointing]] as part of your batch job script. The interactive session example below requests any node with (2) GPUs v100 type, 1 core, 1 GB of memory and 30 minutes of time (default values if not specified) on the ''standard'' partition. |
| |
| <code bash> | <code bash> |
| salloc --partition=standard --gres=gpu:v100 | salloc --partition=standard --gres=gpu:v100:2 |
| </code> | </code> |
| |
| to allocate any type of GPU node available for your interactive job in the standard partition. | If you are unsure of the GPU types and counts available in the ''standard'' partition, see [[abstract:caviness:caviness#compute-nodes|Compute Nodes]] on Caviness. |
| ==== Enhanced Local Scratch nodes ==== | ==== Enhanced Local Scratch nodes ==== |
| |
| |
| |
| ==== Array jobs ==== | ===== Array Jobs ===== |
| |
| An array job essentially runs the same job by generating a new repeated task many times. Each time, the environment variable **SLURM_ARRAY_TASK_ID** is set to a unique value and its value provides input to the job submission script. | An array job essentially runs the same job by generating a new repeated task many times. Each time, the environment variable **SLURM_ARRAY_TASK_ID** is set to a unique value and its value provides input to the job submission script. |
| %%--%%array=1,2,5,19,27 | %%--%%array=1,2,5,19,27 |
| </WRAP> | </WRAP> |
| | |
| | <note important>The default job array size limits are set to 10000 for Slurm on Caviness to avoid oversubscribing the scheduler node's own resource limits (causing scheduling to become sluggish or even unresponsive). See the [[technical:slurm:caviness:arraysize-and-nodecounts#job-array-size-limits|technical explanation]] for why this is necessary. |
| | </note> |
| |
| For more details and information see [[abstract:caviness:runjobs:schedule_jobs#array-jobs1|Array Jobs]]. | For more details and information see [[abstract:caviness:runjobs:schedule_jobs#array-jobs1|Array Jobs]]. |
| ==== Chaining jobs ==== | ===== Chaining Jobs ===== |
| |
| If you have a multiple jobs where you want to automatically run other job(s) after the execution of another job, then you can use chaining. When you chain jobs, remember to check the status of the other job to determine if it successfully completed. This will prevent the system from flooding the scheduler with failed jobs. Here is a simple chaining example with three job scripts ''doThing1.qs'', ''doThing2.qs'' and ''doThing3.qs''. | If you have a multiple jobs where you want to automatically run other job(s) after the execution of another job, then you can use chaining. When you chain jobs, remember to check the status of the other job to determine if it successfully completed. This will prevent the system from flooding the scheduler with failed jobs. Here is a simple chaining example with three job scripts ''doThing1.qs'', ''doThing2.qs'' and ''doThing3.qs''. |
| Four sub-tasks are executed, numbered from 1 through 4. The starting index must be greater than zero, and the ending index must be greater than or equal to the starting index. The //step size// going from one index to the next defaults to one, but can be any positive integer greater than zero. A step size is appended to the sub-task range as in ''2-20:2'' -- proceed from 2 up to 20 in steps of 2, e.g. 2, 4, 6, 8, 10, et al. | Four sub-tasks are executed, numbered from 1 through 4. The starting index must be greater than zero, and the ending index must be greater than or equal to the starting index. The //step size// going from one index to the next defaults to one, but can be any positive integer greater than zero. A step size is appended to the sub-task range as in ''2-20:2'' -- proceed from 2 up to 20 in steps of 2, e.g. 2, 4, 6, 8, 10, et al. |
| |
| <note important>The default [[technical:slurm:caviness/arraysize-and-nodecounts#job-array-size-limits|job array size limits]] for Slurm are used on Caviness to avoid oversubscribing the scheduler node's own resource limits (causing scheduling to become sluggish or even unresponsive). | <note important>The default job array size limits are set to 10000 for Slurm on Caviness to avoid oversubscribing the scheduler node's own resource limits (causing scheduling to become sluggish or even unresponsive). See the [[technical:slurm:caviness:arraysize-and-nodecounts#job-array-size-limits|technical explanation]] for why this is necessary. |
| </note> | </note> |
| ==== Partitioning Job Data ==== | ==== Partitioning Job Data ==== |