Both sides previous revision Previous revision Next revision | Previous revisionLast revisionBoth sides next revision |
abstract:farber:filesystems:filesystems [2018-02-13 12:08] – [Workgroup storage] sraskar | abstract:farber:filesystems:filesystems [2021-03-24 11:44] – [Workgroup] anita |
---|
====== Using Farber's filesystems ====== | ====== Farber Storage ====== |
| |
===== Permanent filesystem ===== | ===== Permanent filesystem ===== |
Each user has 20 GB of disk storage reserved for personal use on the home file system. Users' home directories are in /home (e.g., ''/home/1005''), and the directory name is put in the environment variable ''$HOME'' at login. | Each user has 20 GB of disk storage reserved for personal use on the home file system. Users' home directories are in /home (e.g., ''/home/1005''), and the directory name is put in the environment variable ''$HOME'' at login. |
The permanent file system is configured to allow nearly instantaneous, consistent snapshots. The snapshot contains the original version of the file system, and the live file system contains any changes made since the snapshot was taken. In addition, | The permanent file system is configured to allow nearly instantaneous, consistent snapshots. The snapshot contains the original version of the file system, and the live file system contains any changes made since the snapshot was taken. In addition, |
all your files are regularly replicated at UD's off-campus disaster recovery site. You can use read-only [[/clusters/farber/filesystems?&#home-and-workgroup-snapshots|snapshots]] to revert a previous version, or request to have your files restored from the disaster recovery site. | all your files are regularly replicated at UD's off-campus disaster recovery site. You can use read-only [[:abstract:farber:filesystems:filesystems#home-and-workgroup-snapshots|snapshots]] to revert a previous version, or request to have your files restored from the disaster recovery site. |
| |
You can check to see the size and usage of your home directory with the command | You can check to see the size and usage of your home directory with the command |
==== Workgroup storage ==== | ==== Workgroup storage ==== |
| |
Each research group has at least 1000 GB of shared group ([[abstract:farber:system_access:compute_environment#using-workgroup-and-directories|workgroup]]) storage in the ''/home/work'' directory identified by the <<//investing_entity//>> (e.g., ''/home/work/it_css'') and is referred to as your workgroup directory. This is used for input files, supporting data files, work files, and output files, source code and executables that need to be shared with your research group. | Each research group has at least 1000 GB of shared group ([[abstract:farber:app_dev:compute_env#using-workgroup-and-directories|workgroup]]) storage in the ''/home/work'' directory identified by the <<//investing_entity//>> (e.g., ''/home/work/it_css'') and is referred to as your workgroup directory. This is used for input files, supporting data files, work files, and output files, source code and executables that need to be shared with your research group. |
Just as your home directory, read-only snapshots of workgroup's files are made several times for the passed week. In addition, the filesystem is replicated on UD's off-campus disaster recovery site. [[abstract:farber:filesystems:filesystems#home-and-workgroup-snapshots|Snapshots]] are user-accessible, and older files may be retrieved by special request. | Just as your home directory, read-only snapshots of workgroup's files are made several times for the passed week. In addition, the filesystem is replicated on UD's off-campus disaster recovery site. [[abstract:farber:filesystems:filesystems#home-and-workgroup-snapshots|Snapshots]] are user-accessible, and older files may be retrieved by special request. |
| |
space. This is used for temporary input files, supporting data files, work files, and output files associated with computational tasks run on the cluster. The filesystem is accessible to all of the processor cores via 56 Gbps (FDR) Infiniband. | space. This is used for temporary input files, supporting data files, work files, and output files associated with computational tasks run on the cluster. The filesystem is accessible to all of the processor cores via 56 Gbps (FDR) Infiniband. |
| |
<note warning>Source code and executables must be stored in and executed from Home (''$HOME'') or Workgroup (''$WORKDIR'') storage. No executables are not permitted to run from the Lustre filesystem.</note> | <note warning>Source code and executables must be stored in and executed from Home (''$HOME'') or Workgroup (''$WORKDIR'') storage. No executables are permitted to run from the Lustre filesystem.</note> |
| |
The Lustre filesystem is not backed up. However, it is a robust RAID-6 system. Thus, the filesystem | The Lustre filesystem is not backed up. However, it is a robust RAID-6 system. Thus, the filesystem |
<note important>Remember all of the Lustre filesystem is temporary disk storage. Your workflow should start | <note important>Remember all of the Lustre filesystem is temporary disk storage. Your workflow should start |
by copying needed data files to the high performance Lustre filesystem, ''/lustre/scratch'', and finish by copying results back to your private ''/home'' or shared ''/home/work'' directory. Please clean up (delete) all of the | by copying needed data files to the high performance Lustre filesystem, ''/lustre/scratch'', and finish by copying results back to your private ''/home'' or shared ''/home/work'' directory. Please clean up (delete) all of the |
remaining files in ''/lustre/scratch'' no longer needed. If you do not clean up properly, then files will be purged from ''/lustre/scratch'' by the regular cleanup procedures.</note> | remaining files in ''/lustre/scratch'' no longer needed by using the custom [[abstract:farber:filesystems:lustre#lustre-utilities|Lustre utilities]]. If you do not clean up properly, then files will be purged from ''/lustre/scratch'' by the regular cleanup procedures.</note> |
| |
| **Note**: A full filesystem inhibits use for everyone. |
===== Local filesystem ===== | ===== Local filesystem ===== |
| |
Each compute node has its own 500GB local hard drive, which is needed for time-critical tasks such as managing virtual memory. The system usage of the local disk is kept as small as possible to allow some local disk for your applications, running on the node. | Each compute node has its own 500GB local hard drive, which is needed for time-critical tasks such as managing virtual memory. The system usage of the local disk is kept as small as possible to allow some local disk for your applications, running on the node. |
| |
| ===== Filesystem mount points ===== |
| |
| A //mount point// is the term that describes where the computer puts the files in a hierarchical file system on the cluster. All the filesystems appear as one unified filesystem structure. Here is a list of mount points on Farber: |
| |
| ^ Mount point ^ Backed up ^Description ^ |
| | ''/home'' | yes | Permanent storage, available to all nodes. The location for each user's home directory. A per-user quota system constrains each user. | |
| | ''/lustre'' | no | High-performance storage available to all nodes. May be used for group and for individual use. Appropriate space allocation is controlled by agreed-upon published policies. | |
| | ''/scratch'' | no | A local storage disk only accessible by a single node. Typically used by applications running on that node and having extensive I/O. It is your responsibility to remove files left on that filesystem at the end of your job if a job scheduler hasn't removed them automatically. | |
===== Quotas and usage ===== | ===== Quotas and usage ===== |
| |
Type Path In-use / kiB Available / kiB Pct | Type Path In-use / kiB Available / kiB Pct |
----- -------------------------- ------------ ------------ ---- | ----- -------------------------- ------------ ------------ ---- |
user /home/1006 1691648 20971520 8% | user /home/1201 1691648 20971520 8% |
group /home/work/it_css 39649280 1048576000 4% | group /home/work/it_css 39649280 1048576000 4% |
| |
<code> | <code> |
[traine@farber ~]$ workgroup -g it_css | [traine@farber ~]$ workgroup -g it_css |
[(it_css:dnairn)@farber ~]$ df -h $WORKDIR | [(it_css:traine)@farber ~]$ df -h $WORKDIR |
Filesystem Size Used Avail Use% Mounted on | Filesystem Size Used Avail Use% Mounted on |
storage-nfs1:/export/work/it_css | storage-nfs1:/export/work/it_css |
==== Lustre ==== | ==== Lustre ==== |
| |
All of Lustre is considered scratch storage and subject to removal if necessary for Lustre-performance reasons. All users can create their own directories under the ''/lustre/scratch'' directory and manage them using custom [[:general:filesystems:Lustre-utilities|Lustre utilities]]. To check Lustre usage, use ''df -h /lustre''. | All of Lustre is considered scratch storage and subject to removal if necessary for Lustre-performance reasons. All users can create their own directories under the ''/lustre/scratch'' directory and manage them using custom [[abstract:farber:filesystems:lustre#lustre-utilities|Lustre utilities]]. To check Lustre usage, use ''df -h /lustre''. |
| |
The example below is based on user ''traine'' in workgroup ''it_css'' showing 5.9 GB used from a total filesystem size of 257 TB available on Lustre. | The example below is based on user ''traine'' in workgroup ''it_css'' showing 5.9 GB used from a total filesystem size of 257 TB available on Lustre. |
</note> | </note> |
| |
<note warning>Please use custom [[:general:filesystems:Lustre-utilities|Lustre utilities]] to remove files and check disk usage on the Lustre filesystem .</note> | <note warning>Please use custom [[abstract:farber:filesystems:lustre#lustre-utilities|Lustre utilities]] to remove files and check disk usage on the Lustre filesystem .</note> |
| |
==== Node scratch ==== | ==== Node scratch ==== |
| |
| |
==== Summary of usage recommendations ==== | ==== Usage Recommendations ==== |
| |
[[#recommended-practices|Recommendations:]] | |
| |
**Home directory**: Use your [[#home|home]] directory to store private files. Application software you use will often store its configuration, history and | **Home directory**: Use your [[#home|home]] directory to store private files. Application software you use will often store its configuration, history and |
cache files in your home directory. Generally, keep this directory free and use it for files needed to configure your environment. For example, add [[http://en.wikipedia.org/wiki/Symbolic_link#POSIX_and_Unix-like_operating_systems|symbolic links]] in your home directory to point to files in any of the other directory. The ''/home/'' filesystem is backed-up with [[#home-workgroup-snapshots|snapshots]]. | cache files in your home directory. Generally, keep this directory free and use it for files needed to configure your environment. For example, add [[http://en.wikipedia.org/wiki/Symbolic_link#POSIX_and_Unix-like_operating_systems|symbolic links]] in your home directory to point to files in any of the other directory. The ''/home/'' filesystem is backed-up with [[#home-workgroup-snapshots|snapshots]]. |
| |
**Workgroup directory**: Use the [[#workgroup|workgroup]] directory (/home/work/<<//investing_entity//>>) to build applications for you or your group to use as well as important data, modified source or any other files need to be shared by your research group. See the [[:general/userguide/05_development|Application development]] section for information on building applications. You should create a VALET package for your fellow researchers to access applications you want to share. A typical workflow is to copy the files needed from ''/home/work'' to ''/lustre/scratch'' for the actual run. The ''/home/work'' system is backed-up with [[#home-workgroup-snapshots|snapshots]]. | **Workgroup directory**: Use the [[#workgroup|workgroup]] directory (/home/work/<<//investing_entity//>>) to build applications for you or your group to use as well as important data, modified source or any other files need to be shared by your research group. See the [[:abstract:farber:app_dev:app_dev|Application development]] section for information on building applications. You should create a VALET package for your fellow researchers to access applications you want to share. A typical workflow is to copy the files needed from ''/home/work'' to ''/lustre/scratch'' for the actual run. The ''/home/work'' system is backed-up with [[#home-workgroup-snapshots|snapshots]]. |
| |
**Public scratch directory**: Use the public [[#lustre|Lustre]] scratch directory (/lustre/scratch) for files where high performance is required. Store files produced as intermediate work files, and remove them when your current project is done. That will free up the public scratch workspace others also need. This is also a good place for sharing files and | **Public scratch directory**: Use the public [[#lustre|Lustre]] scratch directory (/lustre/scratch) for files where high performance is required. Store files produced as intermediate work files, and remove them when your current project is done. That will free up the public scratch workspace others also need. This is also a good place for sharing files and |
and data with all users. Files in this directory are not backed up, and subject to removal. Use [[#lustre-utilities|Lustre utilities]] from a compute node to check disk usage and remove files no longer needed. | and data with all users. Files in this directory are not backed up, and subject to removal. Use [[abstract:farber:filesystems:lustre#lustre-utilities|Lustre utilities]] from a compute node to check disk usage and remove files no longer needed. |
| |
**Node scratch directory**: Use the [[#node-scratch|node scratch]] directory (/scratch) for temporary files. The job scheduler software (Grid Engine) creates a subdirectory in /scratch specifically for each job's temporary files. This is done on each node assigned to the job. When the job is complete, the subdirectory and its contents are deleted. This process automatically frees up the local scratch storage that others may need. Files in node scratch directories are not available to the head node, or other compute nodes. | **Node scratch directory**: Use the [[#node-scratch|node scratch]] directory (/scratch) for temporary files. The job scheduler software (Grid Engine) creates a subdirectory in /scratch specifically for each job's temporary files. This is done on each node assigned to the job. When the job is complete, the subdirectory and its contents are deleted. This process automatically frees up the local scratch storage that others may need. Files in node scratch directories are not available to the head node, or other compute nodes. |
**Lustre workgroup directory**: Use the [[#lustre|Lustre]] work directory (/lustre/work/<<//investing_entity//>>) if purchased by your research group for files where high performance is required. Keep just the files needed for | **Lustre workgroup directory**: Use the [[#lustre|Lustre]] work directory (/lustre/work/<<//investing_entity//>>) if purchased by your research group for files where high performance is required. Keep just the files needed for |
your job such as scripts and large data sets used for input or created for output. Remember the disk is not backed up and subject to removal only if needed, so be prepared to rebuild the files if necessary. With batch jobs, the queue script is a | your job such as scripts and large data sets used for input or created for output. Remember the disk is not backed up and subject to removal only if needed, so be prepared to rebuild the files if necessary. With batch jobs, the queue script is a |
record of what you did, but for interactive work, you need to take notes as a record of your work. Use [[#lustre-utilities|Lustre utilities]] from a compute node to check disk usage and remove files no longer needed. | record of what you did, but for interactive work, you need to take notes as a record of your work. Use [[abstract:farber:filesystems:lustre#lustre-utilities|Lustre utilities]] from a compute node to check disk usage and remove files no longer needed. |