technical:slurm:darwin:swap-control-implementation

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		 Previous revision
technical:slurm:darwin:swap-control-implementation [2021-11-19 12:55] – created freytechnical:slurm:darwin:swap-control-implementation [2021-12-03 10:28] (current) – [Issues] frey
Line 1: Line 1:
 +====== Revisions to Slurm Configuration v1.0.7 on DARWIN ======
 +
 +This document summarizes alterations to the Slurm job scheduler configuration on the DARWIN cluster.
 +
 +===== Issues =====
 +
 +See [[technical:slurm:swap-control|this document]] discussing swap limits in Slurm jobs.
 +
 +===== Implementation =====
 +
 +  * The ''dynamic_swap_limits'' SPANK plugin will be compiled and installed
 +  * The ''plugstack.conf'' configuration file will be modified to require the ''dynamic_swap_limits'' plugin
 +
 +The following aspects of the system construction will be observed in the configuration of the plugin:
 +
 +  * In all cases, the active swap present on the node will be used as the ''max_swap'' value
 +  * Most nodes consist of 64 cores; 1/64 = 1.5625%, so by default jobs will be granted 1% of ''max_swap'' per CPU
 +  * The V100 nodes consist of 48 cores; 1/48 = 2.0833%, so by default jobs will be granted 1.5% of ''max_swap'' per CPU
 +  * The Mi100 node consists of 128 cores; 1/128 = 0.78125%, so by default jobs will be granted 0.25% of ''max_swap'' per CPU
 +  * Jobs running under the //extended-mem// partition on the extended memory node will have no swap limits enforced (jobs are scheduled user-exclusive on this node, so any failures will not impact other users' jobs)
 +
 +These details produce the following configuration string:
 +
 +<code>
 +partition(extended-mem)=none,host(r2v[00-02])=1.5%/cpu,host(r0m01)=0.25%/cpu,default()=1%/cpu
 +</code>
 +
 +===== Impact =====
 +
 +No downtime is expected.  The ''slurmd'' daemon must be restarted on all compute nodes, but currently-executing jobs/job steps should not be affected (they will reconnect to the new ''slurmd'' as necessary to communicate job status, etc.).  The ''slurmctld'' daemons do not use the SPANK plugin, thus they do not need to be restarted.
 +
 +===== Timeline =====
 +
 +^Date ^Time ^Goal/Description ^
 +|2021-11-19| |Authoring of this document|
 +|2021-11-24|09:00|Implementation|
 +|2021-12-01|14:54|Update:  appropriate aggregate limits on job cgroup|