This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision |
| technical:slurm:caviness:mandatory_gpu_type [2024-01-24 16:10] – [Issues] anita | technical:slurm:caviness:mandatory_gpu_type [2024-01-30 17:17] (current) – [Issues] anita |
|---|
| In this case, **only the generic implicit limit is affected.** Job 123456789 can use any type of GPU to which workgroup_X has access, but no type-specific limit will influence the scheduler's choice of GPU type. Even if workgroup_X already has running jobs using 2 of 2 A100 GPUs, job 123456789 would be allowed to use a third A100 GPU — effectively borrowing an A100 against the quota of P100 and V100 GPUs they also purchased. This is not the intended behavior. | In this case, **only the generic implicit limit is affected.** Job 123456789 can use any type of GPU to which workgroup_X has access, but no type-specific limit will influence the scheduler's choice of GPU type. Even if workgroup_X already has running jobs using 2 of 2 A100 GPUs, job 123456789 would be allowed to use a third A100 GPU — effectively borrowing an A100 against the quota of P100 and V100 GPUs they also purchased. This is not the intended behavior. |
| |
| <note important>**[VERY IMPORTANT]** Once this change goes into effect all job scripts and command line requests for GPUs using the syntax ''%%--%%gres=gpu'' or ''%%--%%gres=gpu:<<#>>'' must be altered to include the desired GPU type: for example, ''%%--%%gres=gpu:a100'' or ''%%--%%gres=gpu:a100:<<#>>''. The command ''sworkgroup -g <<workgroup>> %%--%%limits'' displays the GPU types and counts available to your workgroup for jobs in the workgroup partition. Jobs submitted to the standard partition (for which workgroup GPU limits do not apply) **must** also specify the GPU type once the change goes into effect.</note> | <note important>**[VERY IMPORTANT]** Once this change goes into effect all job scripts and command line requests for GPUs using the syntax ''%%--%%gres=gpu'' or ''%%--%%gres=gpu:<<#>>'' **must** be altered to include the desired GPU type: for example, ''%%--%%gres=gpu:a100'' or ''%%--%%gres=gpu:a100:<<#>>''. The command ''sworkgroup -g <<workgroup>> %%--%%limits'' displays the GPU types and counts available to your workgroup for jobs in the workgroup partition. Jobs submitted to the standard partition (for which workgroup GPU limits do not apply) **must** also specify the GPU type once the change goes into effect. If you are unsure of the GPU types and counts available in the ''standard'' partition, see [[abstract:caviness:caviness#compute-nodes|Compute Nodes]] on Caviness.</note> |
| |
| ==== Generational Change ==== | ==== Generational Change ==== |
| [user@login00.caviness ~]$ sbatch --gres gpu:a100:2 --partition workgroup_X … | [user@login00.caviness ~]$ sbatch --gres gpu:a100:2 --partition workgroup_X … |
| </code> | </code> |
| | |
| | If you are unsure of the GPU types and counts available in your workgroup partition, use the command ''sworkgroup -g workgroup_X %%--%%limits''. Remember jobs submitted to the ''standard'' partition (for which workgroup GPU limits do not apply) **must** also specify the GPU type. If you are unsure of the GPU types and counts available in the ''standard'' partition, see [[abstract:caviness:caviness#compute-nodes|Compute Nodes]] on Caviness. |
| |
| ===== Impact ===== | ===== Impact ===== |