| Both sides previous revision Previous revision Next revision | Previous revision |
| abstract:caviness:runjobs:schedule_jobs [2024-01-30 17:10] – [GPU nodes] anita | abstract:caviness:runjobs:schedule_jobs [2024-05-23 11:32] (current) – [Memory] anita |
|---|
| where '<N>' should be the number of GPUs and depends on the specific [[abstract:caviness:caviness#compute-nodes|compute node specifications]]. | where '<N>' should be the number of GPUs and depends on the specific [[abstract:caviness:caviness#compute-nodes|compute node specifications]]. |
| |
| <note important>**VERY IMPORTANT:** Keep in mind that not all memory can be reserved for a node due to a small amount required for system use. As a result, the maximum amount of memory that can be specified is based on what Slurm shows as available. For example, the baseline nodes in Caviness show a memory size of 124 GiB versus the 128 GiB of physical memory present in them. This means if you try to specify the full amount of memory (i.e. 128G), then Slurm will try to run the job on a larger memory node as long as you have access to a larger memory node. This will work if you specify the standard partition or if you specify a workgroup partition and your research group purchased a larger memory node, otherwise your job will never run.</note> | <note important>**VERY IMPORTANT:** Keep in mind that not all memory can be reserved for a node due to a small amount required for system use. As a result, the maximum amount of memory that can be specified is based on what Slurm shows as available. For example, the baseline nodes in Caviness show a memory size of 124 GiB versus the 128 GiB of physical memory present in them. This means if you try to specify the full amount of memory (i.e. 128G), then Slurm will try to run the job on a larger memory node as long as you have access to a larger memory node. This will work if you specify the standard partition or if you specify a workgroup partition and your research group purchased a larger memory node, otherwise your job will never run. You may also use ''%%--%%mem=0'' to request all the memory on a node.</note> |
| |
| ==== Exclusive access ==== | ==== Exclusive access ==== |
| ==== GPU nodes ==== | ==== GPU nodes ==== |
| |
| After entering into the workgroup, GPU nodes can be requested through an interactive session using ''salloc'' or through batch submission using ''sbatch''. An appropriate partition name (such as a workgroup for running or ''devel'' if you need to compile on a GPU node) and a GPU resource and type has to be mentioned while running the command as below. | After entering into the workgroup, GPU nodes can be requested through an interactive session using ''salloc'' or through batch submission using ''sbatch''. An appropriate partition name (such as a workgroup for running or ''devel'' if you need to compile on a GPU node) and a GPU resource and type **must** be specified while running the command as below. |
| |
| <code bash> | <code bash> |
| </code> | </code> |
| |
| Any user can employ a GPU by running in the ''standard'' partition, however keep in mind a GPU type **must** be specified, jobs can be preempted and would require [[abstract:caviness:runjobs:schedule_jobs#handling-system-signals-aka-checkpointing|checkpointing]] as part of your batch job script. The interactive session example below requests any node with (2) GPUs v100 type, 1 core, 1 GB of memory and 30 minutes of time (default values if not specified) on the standard partition. | Any user can employ a GPU by running in the ''standard'' partition, however keep in mind a GPU type **must** be specified, jobs can be preempted and would require [[abstract:caviness:runjobs:schedule_jobs#handling-system-signals-aka-checkpointing|checkpointing]] as part of your batch job script. The interactive session example below requests any node with (2) GPUs v100 type, 1 core, 1 GB of memory and 30 minutes of time (default values if not specified) on the ''standard'' partition. |
| |
| <code bash> | <code bash> |
| </code> | </code> |
| |
| If you are unsure of what GPU types are available when using the ''standard'' partition, see [[abstract:caviness:caviness#compute-nodes|Compute Nodes]] on Caviness. | If you are unsure of the GPU types and counts available in the ''standard'' partition, see [[abstract:caviness:caviness#compute-nodes|Compute Nodes]] on Caviness. |
| ==== Enhanced Local Scratch nodes ==== | ==== Enhanced Local Scratch nodes ==== |
| |