| Next revisionBoth sides next revision |
| technical:slurm:auto_tmpdir [2020-03-12 10:47] – created frey | technical:slurm:auto_tmpdir [2020-03-12 11:03] – frey |
|---|
| The latter two items have remained an issue, though, since it is up to the software (or even the user's job script) to remove the temporary files when the job completes. Crashes and early termination of Open MPI are particularly annoying because they leave behind many files in ''/dev/shm'', which is not stored on disk but in physical RAM and swap (which diminishes the amount of available RAM on the node over time). It was necessary for us to deploy on Farber and Caviness a periodic cleanup program that identifies which files in ''/dev/shm'' are in-use and delete the rest. | The latter two items have remained an issue, though, since it is up to the software (or even the user's job script) to remove the temporary files when the job completes. Crashes and early termination of Open MPI are particularly annoying because they leave behind many files in ''/dev/shm'', which is not stored on disk but in physical RAM and swap (which diminishes the amount of available RAM on the node over time). It was necessary for us to deploy on Farber and Caviness a periodic cleanup program that identifies which files in ''/dev/shm'' are in-use and delete the rest. |
| |
| ===== Bind Mounts ===== | ==== Bind mounts ==== |
| |
| The Linux virtual file system (VFS)- is a hierarchy of directories and files, referenced by //path//. That //namespace// is a single tree, unlike Windows where each distinct storage component is assigned a letter (your primary hard drive is your "C" drive, etc.). In Linux, though, the directories and files on each storage component are //mounted// at some position within the single VFS tree. On Caviness, IT-maintained software can be found under ''/opt/shared'' on every node, yet those directories and files are not stored in disks present in that node; rather, they are stored on a file server whose storage is //mounted// at ''/opt/shared'' on the nodes. Interacting with the ''/opt'' directory requires Linux to access a local disk in the node, but moving to ''/opt/shared'' requires interaction with a file server across the network, all transparent to the user who is accessing the file system. | The Linux virtual file system (VFS)- is a hierarchy of directories and files, referenced by //path//. That //namespace// is a single tree, unlike Windows where each distinct storage component is assigned a letter (your primary hard drive is your "C" drive, etc.). In Linux, though, the directories and files on each storage component are //mounted// at some position within the single VFS tree. On Caviness, IT-maintained software can be found under ''/opt/shared'' on every node, yet those directories and files are not stored in disks present in that node; rather, they are stored on a file server whose storage is //mounted// at ''/opt/shared'' on the nodes. Interacting with the ''/opt'' directory requires Linux to access a local disk in the node, but moving to ''/opt/shared'' requires interaction with a file server across the network, all transparent to the user who is accessing the file system. |
| Once job 8456 has altered what's mounted at ''/tmp'', job 8451 will no longer see the temporary files it created in ''/tmp/job-8451'' and the program will likely crash. The same problem would exist for Slurm and OS programs that were using files in ''/tmp'' prior to job 8451's executing. | Once job 8456 has altered what's mounted at ''/tmp'', job 8451 will no longer see the temporary files it created in ''/tmp/job-8451'' and the program will likely crash. The same problem would exist for Slurm and OS programs that were using files in ''/tmp'' prior to job 8451's executing. |
| |
| ===== Namespaces to the Rescue ===== | ==== Namespaces to the rescue ==== |
| |
| For the bind-mount solution to work, each Slurm job needs to have its own VFS tree that is independent of other programs on the node. Linux //mount namespaces// are exactly that: | For the bind-mount solution to work, each Slurm job needs to have its own VFS tree that is independent of other programs on the node. Linux //mount namespaces// are exactly that: |
| |
| With the same procedure applied to ''/dev/shm'', the major sources of orphaned temporary files would be contained. | With the same procedure applied to ''/dev/shm'', the major sources of orphaned temporary files would be contained. |
| | |
| | ===== The auto_tmpdir plugin ===== |
| | |
| | |