| Both sides previous revision Previous revision Next revision | Previous revision |
| abstract:caviness:runjobs:schedule_jobs [2024-01-30 17:18] – [GPU nodes] anita | abstract:caviness:runjobs:schedule_jobs [2024-09-25 14:10] (current) – [Array Jobs] anita |
|---|
| where '<N>' should be the number of GPUs and depends on the specific [[abstract:caviness:caviness#compute-nodes|compute node specifications]]. | where '<N>' should be the number of GPUs and depends on the specific [[abstract:caviness:caviness#compute-nodes|compute node specifications]]. |
| |
| <note important>**VERY IMPORTANT:** Keep in mind that not all memory can be reserved for a node due to a small amount required for system use. As a result, the maximum amount of memory that can be specified is based on what Slurm shows as available. For example, the baseline nodes in Caviness show a memory size of 124 GiB versus the 128 GiB of physical memory present in them. This means if you try to specify the full amount of memory (i.e. 128G), then Slurm will try to run the job on a larger memory node as long as you have access to a larger memory node. This will work if you specify the standard partition or if you specify a workgroup partition and your research group purchased a larger memory node, otherwise your job will never run.</note> | <note important>**VERY IMPORTANT:** Keep in mind that not all memory can be reserved for a node due to a small amount required for system use. As a result, the maximum amount of memory that can be specified is based on what Slurm shows as available. For example, the baseline nodes in Caviness show a memory size of 124 GiB versus the 128 GiB of physical memory present in them. This means if you try to specify the full amount of memory (i.e. 128G), then Slurm will try to run the job on a larger memory node as long as you have access to a larger memory node. This will work if you specify the standard partition or if you specify a workgroup partition and your research group purchased a larger memory node, otherwise your job will never run. You may also use ''%%--%%mem=0'' to request all the memory on a node.</note> |
| |
| ==== Exclusive access ==== | ==== Exclusive access ==== |
| |
| |
| ==== Array jobs ==== | ===== Array Jobs ===== |
| |
| An array job essentially runs the same job by generating a new repeated task many times. Each time, the environment variable **SLURM_ARRAY_TASK_ID** is set to a unique value and its value provides input to the job submission script. | An array job essentially runs the same job by generating a new repeated task many times. Each time, the environment variable **SLURM_ARRAY_TASK_ID** is set to a unique value and its value provides input to the job submission script. |
| %%--%%array=1,2,5,19,27 | %%--%%array=1,2,5,19,27 |
| </WRAP> | </WRAP> |
| | |
| | <note important>The default job array size limits are set to 10000 for Slurm on Caviness to avoid oversubscribing the scheduler node's own resource limits (causing scheduling to become sluggish or even unresponsive). See the [[technical:slurm:caviness:arraysize-and-nodecounts#job-array-size-limits|technical explanation]] for why this is necessary. |
| | </note> |
| |
| For more details and information see [[abstract:caviness:runjobs:schedule_jobs#array-jobs1|Array Jobs]]. | For more details and information see [[abstract:caviness:runjobs:schedule_jobs#array-jobs1|Array Jobs]]. |
| ==== Chaining jobs ==== | ===== Chaining Jobs ===== |
| |
| If you have a multiple jobs where you want to automatically run other job(s) after the execution of another job, then you can use chaining. When you chain jobs, remember to check the status of the other job to determine if it successfully completed. This will prevent the system from flooding the scheduler with failed jobs. Here is a simple chaining example with three job scripts ''doThing1.qs'', ''doThing2.qs'' and ''doThing3.qs''. | If you have a multiple jobs where you want to automatically run other job(s) after the execution of another job, then you can use chaining. When you chain jobs, remember to check the status of the other job to determine if it successfully completed. This will prevent the system from flooding the scheduler with failed jobs. Here is a simple chaining example with three job scripts ''doThing1.qs'', ''doThing2.qs'' and ''doThing3.qs''. |
| Four sub-tasks are executed, numbered from 1 through 4. The starting index must be greater than zero, and the ending index must be greater than or equal to the starting index. The //step size// going from one index to the next defaults to one, but can be any positive integer greater than zero. A step size is appended to the sub-task range as in ''2-20:2'' -- proceed from 2 up to 20 in steps of 2, e.g. 2, 4, 6, 8, 10, et al. | Four sub-tasks are executed, numbered from 1 through 4. The starting index must be greater than zero, and the ending index must be greater than or equal to the starting index. The //step size// going from one index to the next defaults to one, but can be any positive integer greater than zero. A step size is appended to the sub-task range as in ''2-20:2'' -- proceed from 2 up to 20 in steps of 2, e.g. 2, 4, 6, 8, 10, et al. |
| |
| <note important>The default [[technical:slurm:caviness/arraysize-and-nodecounts#job-array-size-limits|job array size limits]] for Slurm are used on Caviness to avoid oversubscribing the scheduler node's own resource limits (causing scheduling to become sluggish or even unresponsive). | <note important>The default job array size limits are set to 10000 for Slurm on Caviness to avoid oversubscribing the scheduler node's own resource limits (causing scheduling to become sluggish or even unresponsive). See the [[technical:slurm:caviness:arraysize-and-nodecounts#job-array-size-limits|technical explanation]] for why this is necessary. |
| </note> | </note> |
| ==== Partitioning Job Data ==== | ==== Partitioning Job Data ==== |