| Both sides previous revision Previous revision Next revision | Previous revision |
| abstract:caviness:filesystems:filesystems [2021-03-24 11:45] – [Home] anita | abstract:caviness:filesystems:filesystems [2023-01-13 14:04] (current) – [Lustre] anita |
|---|
| ==== Lustre storage==== | ==== Lustre storage==== |
| |
| User storage is available on a [[abstract:caviness:filesystems:lustre|high-performance Lustre-based filesystem]] having 210 TB of usable | User storage is available on a [[abstract:caviness:filesystems:lustre|high-performance Lustre-based filesystem]] having 403TB TB of usable |
| space. This is used for temporary input files, supporting data files, work files, and output files associated with computational tasks run on the cluster. The filesystem is accessible to all of the processor cores via Omni-path Infiniband. | space. This is used for temporary input files, supporting data files, work files, and output files associated with computational tasks run on the cluster. The filesystem is accessible to all of the processor cores via Omni-path Infiniband. The default stripe count is set to 1 and the default striping is a single stripe distributed across all available OSTs on Lustre. See [[https://www.nas.nasa.gov/hecc/support/kb/lustre-best-practices_226.html|Lustre Best Practices]] from Nasa. |
| |
| <note warning>Source code and executables must be stored in and executed from Home (''$HOME'') or Workgroup (''$WORKDIR'') storage. No executables are permitted to run from the Lustre filesystem. There are technical reasons why this is not permissible.</note> | <note warning>Source code and executables must be stored in and executed from Home (''$HOME'') or Workgroup (''$WORKDIR'') storage. No executables are permitted to run from the Lustre filesystem. There are technical reasons why this is not permissible.</note> |
| ==== Node scratch ==== | ==== Node scratch ==== |
| |
| Each compute node has its own 900GB local hard drive, which is needed for time-critical tasks such as managing virtual memory. The system usage of the local disk is kept as small as possible to allow some local disk for your applications, running on the node. | Each compute node has its own 900GB local hard drive (or for enhanced local scratch nodes 32TB), which is needed for time-critical tasks such as managing virtual memory. The system usage of the local disk is kept as small as possible to allow some local disk for your applications, running on the node. |
| |
| ===== Quotas and usage ===== | ===== Quotas and usage ===== |
| |
| </code> | </code> |
| | |
| | <note important>**IMPORTANT**: All users are encouraged to delete files no longer needed once results are gathered and collated. Email notifications are sent to each user when ''$HOME'' is close to or has exceeded their quota. Principle stakeholders are notified via email when ''$WORKDIR'' is close to or has exceeded their quota. Remember all quota issues will likely result in job failures, and especially ''$WORKDIR'' will likely cause jobs to fail for everyone in the workgroup. And of course Lustre is extremely important to clean up. IT will periodically send email to all users or principle stakeholders to clean up to keep ''/lustre/scratch'' below 80%. However if ''/lustre/scratch'' fills up, this will likely cause ALL jobs to fail for everyone on Caviness. |
| | </note> |
| | |
| | <note tip> |
| | Please take the time to periodically cleanup your files in ''$HOME'', ''$WORKDIR'' and ''/lustre/scratch'' by doing so from a compute node. We recommend using the ''devel'' partition for this purpose. Specify your workgroup (e.g. ''workgroup -g it_css'') and use ''salloc --partition=devel'' to put you on a compute node with the default resources (1 core, 1 GB memory, and 30 minutes) to delete unnecessary files. If you think you will need additional resources (like more time), see [[abstract:caviness:runjobs:queues#the-devel-partition|Caviness partitions]] for complete details on max resources allowed to be requested on the ''devel'' partition.</note> |
| | |
| ==== Home ==== | ==== Home ==== |
| Each user's home directory has a hard quota limit of 20 GB. To check usage, use | Each user's home directory has a hard quota limit of 20 GB. To check usage, use |
| All of Lustre is considered scratch storage and subject to removal if necessary for Lustre-performance reasons. All users can create their own directories under the ''/lustre/scratch'' directory and manage them based on understanding the concepts of [[abstract:caviness:filesystems:lustre|Lustre]]. To check Lustre usage, use ''df -h /lustre/scratch''. | All of Lustre is considered scratch storage and subject to removal if necessary for Lustre-performance reasons. All users can create their own directories under the ''/lustre/scratch'' directory and manage them based on understanding the concepts of [[abstract:caviness:filesystems:lustre|Lustre]]. To check Lustre usage, use ''df -h /lustre/scratch''. |
| |
| The example below is based on user ''traine'' in workgroup ''it_css'' showing 40 MB used from a total filesystem size of 191 TB available on Lustre. | The example below is based on user ''traine'' in workgroup ''it_css'' showing 225 TB used from a total filesystem size of 367 TB available on Lustre. |
| |
| <code> | <code> |
| [(it_css:traine)@login01 ~]$ df -h /lustre/scratch | [(it_css:traine)@login01 ~]$ df -h /lustre/scratch |
| Filesystem Size Used Avail Use% Mounted on | Filesystem Size Used Avail Use% Mounted on |
| 10.65.32.18@o2ib:10.65.32.19@o2ib:/scratch 191T 40M 191T 1% /lustre/scratch | 10.65.32.18@o2ib:10.65.32.19@o2ib:/scratch 367T 225T 142T 62% /lustre/scratch |
| |
| </code> | </code> |