| Both sides previous revision Previous revision Next revision | Previous revisionLast revisionBoth sides next revision |
| abstract:darwin:app_dev:compute_env [2021-04-26 10:39] – [Using workgroup and directories] anita | abstract:darwin:app_dev:compute_env [2021-06-22 11:41] – anita |
|---|
| |
| ===== Environment variables ===== | ===== Environment variables ===== |
| |
| ==== Introduction ==== | |
| |
| Environment variables store dynamic system values that affect the user environment. For example, the ''PATH'' environment variable tells the operating system where to look for executables. Many UNIX commands and tools, such as the compilers, debuggers, profilers, editors, and applications with graphical user interfaces, often look at environment variables for information they need to function. The man pages for these programs typically have an ENVIRONMENT VARIABLES section with a list of variable names which tells how the program uses the values. | Environment variables store dynamic system values that affect the user environment. For example, the ''PATH'' environment variable tells the operating system where to look for executables. Many UNIX commands and tools, such as the compilers, debuggers, profilers, editors, and applications with graphical user interfaces, often look at environment variables for information they need to function. The man pages for these programs typically have an ENVIRONMENT VARIABLES section with a list of variable names which tells how the program uses the values. |
| | |
| | This is why we encourage users to use [[abstract:darwin:app_dev:compute_env#using-valet-and-your-unix-environment|VALET]] to modify your environment versus explicitly setting environment variables. |
| |
| In bash, a variable must be exported to be used as an environment variable. By convention, environment variables are all uppercase. You can display a list of currently set environment variables by typing | In bash, a variable must be exported to be used as an environment variable. By convention, environment variables are all uppercase. You can display a list of currently set environment variables by typing |
| Exiting the login session or typing the **logout** command executes your ''~/.bash_logout'' file and terminates your session. Add commands to ''~/.bash_logout'' that you want to execute at logout. | Exiting the login session or typing the **logout** command executes your ''~/.bash_logout'' file and terminates your session. Add commands to ''~/.bash_logout'' that you want to execute at logout. |
| |
| To restore the ''.bash_profile'', ''.bashrc'', ''.bash_udit'' and ''.bash_logout'' files in your home directory to their original state, type: | To restore the ''.bash_profile'', ''.bashrc'', ''.bash_udit'' and ''.bash_logout'' files in your home directory to their original state, type from the login node: |
| |
| <code bash> | <code bash> |
| cp /etc/skel/.bash* $HOME | cp /opt/shared/templates/homedir/.bash* $HOME |
| </code> | </code> |
| | |
| <note tip>**Where to put startup commands:** You can put bash commands in either ''~/.bashrc'' or ''~/.bash_profile''. Here are general suggestions: | <note tip>**Where to put startup commands:** You can put bash commands in either ''~/.bashrc'' or ''~/.bash_profile''. Here are general suggestions: |
| * Even if you have favorite commands from other systems, start by using the supplied files. Only modify ''.bash_udit'' for customization. | * Even if you have favorite commands from other systems, start by using the supplied files and **only** modify ''.bash_udit'' for customization. |
| * Add essential commands that you fully understand, and keep it simple. Quoting rules can be complicated. | * Add essential commands that you fully understand, and keep it simple. Quoting rules can be complicated. |
| * Do not depend on the order of command execution. Do not assume your environment, set in ''.bash_profile'', will be available when the commands in ''.bashrc'' are executed. | * Do not depend on the order of command execution. Do not assume your environment, set in ''.bash_profile'', will be available when the commands in ''.bashrc'' are executed. |
| * Do not include commands that spawn new shells, such as ''workgroup''. | * Do not include commands that spawn new shells, such as ''workgroup''. |
| * Be very careful of commands that may produce output. If you must, only execute them after a test to make sure there is a terminal to receive the output. Keep in mind using any commands that produce output may break other applications like file transfer (sftp, scp, WINscp). | * Be very careful of commands that may produce output. If you must, only execute them after a test to make sure there is a terminal to receive the output. Keep in mind using any commands that produce output may break other applications like file transfer (sftp, scp, WinSCP). |
| | * Do not include VALET commands as they produce output and will be a part of every job submitted which could cause conflicts with other applications you are trying to run in your job script. |
| * Keep a session open on the cluster, so when you make a change that prevents you from logging on you can reverse the last change, or copy the ''/etc/skel'' file to start over. | * Keep a session open on the cluster, so when you make a change that prevents you from logging on you can reverse the last change, or copy the ''/etc/skel'' file to start over. |
| </note> | </note> |
| |
| There are some key environment variables that are set for you, and are important for your work on any cluster. They are used to find directories for your projects. These environment variables are set on initial connection to a cluster, and will be changed if you | There are some key environment variables that are set for you, and are important for your work on any cluster. They are used to find directories for your projects. These environment variables are set on initial connection to a cluster, and will be changed if you |
| * set your workgroup (cluster group //investing-entity// name) with the **workgroup** command, | * set your workgroup (allocation group //allocation_group// name) with the **workgroup** command, |
| * change to your project directory with the **cd** command, | * change to your project directory with the **cd** command, |
| * connect to the compute node resources with **salloc** (or **sbatch**) command. | * connect to the compute node resources with **salloc** (or **sbatch**) command specifying a single partition your allocation workgroup has access based on resources requested for your allocation. |
| |
| === Connecting to login node === | === Connecting to login node === |
| </note> | </note> |
| <note important>Clusters may be configured to have multiple login nodes, with one common name for connecting. For example, on the DARWIN cluster, the hostname may be set to ''login00'' or ''login01'', and the standard prompt and window title bar will indicate which login node on //darwin//. </note> | <note important>Clusters may be configured to have multiple login nodes, with one common name for connecting. For example, on the DARWIN cluster, the hostname may be set to ''login00'' or ''login01'', and the standard prompt and window title bar will indicate which login node on //darwin//. </note> |
| | |
| === Setting workgroup === | === Setting workgroup === |
| |
| |
| ^ Variable ^ Example Value ^ Description ^ | ^ Variable ^ Example Value ^ Description ^ |
| | ''WORKDIR'' | ''/lustre/it_css'' | Allocation group private work directory, this is not writeable | | | ''WORKDIR'' | ''/lustre/it_css'' | Allocation workgroup directory, this is not writeable | |
| | ''WORKGROUP'' | ''it_css'' | Current workgroup name | | | ''WORKGROUP'' | ''it_css'' | Current allocation workgroup name | |
| | ''WORKDIR_USER'' | ''/lustre/it_css/users/<<//uid//>>'' | Allocation group private user workgroup directory | | | ''WORKDIR_USER'' | ''/lustre/it_css/users/<<//uid//>>'' | Allocation workgroup user directory | |
| | ''WORKDIR_SW'' | ''/lustre/it_css/sw'' | Allocation group software workgroup directory | | | ''WORKDIR_SW'' | ''/lustre/it_css/sw'' | Allocation workgroup software directory | |
| |
| <note tip>Use ''$WORKDIR'' when you refer to your allocation group private directory. This will improve portability. </note> | <note tip>Use specific environment variables such as ''$WORKDIR_USERS'' when referring to your allocation workgroup user directory and ''$WORKDIR_SW'' when referring to your allocation workgroup software directory. This will improve portability. </note> |
| <note tip> | <note tip> |
| It is always important to be aware of your current workgroup name. The standard prompt includes the workgroup name, added to your username and host. You must have a workgroup name in your prompt to use that allocations group's compute node resources to submit jobs using [[[[abstract/darwin/runjobs/schedule_jobs#batch-jobs-qsub|sbatch]] or salloc. An example prompt after the **workgroup** command, | It is always important to be aware of your current allocation workgroup name. The standard prompt includes the allocation workgroup name, added to your username and host. You must have an allocation workgroup name in your prompt to use that allocation group's compute node resources to [[abstract/darwin/runjobs/schedule_jobs|submit jobs]] using ''sbatch'' or ''salloc''. An example prompt after the **workgroup** command, |
| |
| ''[(it_css:traine)@login01.darwin ~]$ '' | ''[(it_css:traine)@login01.darwin ~]$ '' |
| === Changing directory === | === Changing directory === |
| |
| When you first connect to the login node, all your commands are executed from your home directory. Most of your work will be done in your private allocation group work directory. The **workgroup** command has an option to start you in the private group work directory. For example, | When you first connect to the login node, all your commands are executed from your home directory (''~''). Most of your work will be done in your allocation workgroup directory. The **workgroup** command has an option to start you in the allocation workgroup work directory. For example, |
| <code bash> | <code bash> |
| workgroup -cg it_css | workgroup -cg it_css |
| cd $WORKDIR_USERS/project/fuelcell | cd $WORKDIR_USERS/project/fuelcell |
| </code> | </code> |
| The first is using a path name relative to the current working directory (implied ''./''). The second and third use the full path (''$WORKDIR'' and ''$WORKDIR_USERS'' always begins with a ''/''). In all cases the directory is changed, and the ''$PWD'' environment variable is set: | The first is using a path name relative to the current working directory (implied ''./''). The second, third and fourth use the full path (''$WORKDIR'' and ''$WORKDIR_USERS'' always begins with a ''/''). In all cases the directory is changed, and the ''$PWD'' environment variable is set: |
| |
| ^ Variable ^ Example Value ^ Description ^ | ^ Variable ^ Example Value ^ Description ^ |
| |
| <note tip> | <note tip> |
| It is always important to be aware of your current working directory. The standard prompt ends with the basename of ''PWD''. In these two examples the basename is the same, ''traine'', but the standard bash ''PROMPT_COMMAND'', which is executed every time you change directories, will put the full path of your current working directory in your window title. For example, | It is always important to be aware of your current working directory. The standard prompt ends with the basename of ''PWD''. In these two examples the basename is the same, ''1201'', but the standard bash ''PROMPT_COMMAND'', which is executed every time you change directories, will put the full path of your current working directory in your window title. For example, |
| |
| ''traine@login00.darwin:/lustre/it_css/users/1006'' | ''traine@login00.darwin:/lustre/it_css/users/1201'' |
| </note> | </note> |
| |
| ^ Variable ^ Example Value ^ Description ^ | ^ Variable ^ Example Value ^ Description ^ |
| | ''HOSTNAME'' | ''r00n17'' | Compute node name | | | ''HOSTNAME'' | ''r00n17'' | Compute node name | |
| | ''TMPDIR'' | ''/tmp/1185/0'' | Temporary disk space | | | ''TMPDIR'' | ''/tmp'' | Temporary disk space | |
| |
| An empty directory is created by the SLURM job scheduler that is associated with your job and defined as ''TMPDIR''. This is a safe place to store temporary files that will not interfere with other jobs and tasks you or other members of your group may be executing. This directory is automatically emptied on normal termination of your job. This way the usage on the node scratch filesystem will not grow over time. | An empty directory is created by the SLURM job scheduler that is associated with your job and defined as ''TMPDIR''. This is a safe place to store temporary files that will not interfere with other jobs and tasks you or other members of your group may be executing. This directory is automatically emptied on normal termination of your job. This way the usage on the node scratch filesystem will not grow over time. |
| |
| <note important>Before submitting jobs you **must** first use the **workgroup** command. Type ''workgroup -h'' for additional information. | <note important>Before submitting jobs you **must** first use the **workgroup** command. Type ''workgroup -h'' for additional information. |
| | Both **sbatch** and **salloc** will start in the same project directory you set on the login node and will require a single [[abstract:darwin:runjobs:queues|partition]] to be specified to be able to submit a batch or interactive session. |
| Every time you connect to the login node, you should use **workgroup** to set the allocation group you will being working in, and then **cd** to change to the directory for the project you will working on. Both **sbatch** and **salloc** will start in the same project directory you set on the login node. | |
| </note> | </note> |
| |
| | ''man valet'' | Complete documentation of VALET commands. | | | ''man valet'' | Complete documentation of VALET commands. | |
| |
| ==== Having a personal Work directory apart from the common workgroup directory ==== | |
| |
| Users can maintain their own work directories (''$WORKDIR'') apart from the common work directory of the workgroup. It is also possible to make VALET software to look into this user level work directory. Users can now add ''~/.workdirs.yaml'' to override what gets set for ''$WORKDIR'' for workgroups to which they are members. Detailed explanation follows. | |
| |
| <code bash> | |
| [traine@login00 ~]$ workdir -g it_css | |
| /work/it_css | |
| |
| [traine@login00 ~]$ echo "it_css: /work/it_css/users/traine" > ~/.workdirs.yaml | |
| |
| [traine@login00 ~]$ workdir -g it_css | |
| /work/it_css/users/traine | |
| </code> | |
| |
| Now inside the new work directory, users can create sw/valet directory to add valet packages | |
| |
| <code bash> | |
| [traine@login00 ~]$ ls -l /work/it_css/users/traine/sw/valet | |
| total 11 | |
| -rw-r--r-- 1 frey it_nss 480 Sep 21 16:06 wrf.vpkg_yaml | |
| </code> | |
| |
| From this point, valet looks at this directory for possible valet packages and also the ''$WORKDIR'' after changing to workgroup will point to the path overridden in ''workdirs.yaml'' file. | |
| |
| <code bash> | |
| traine@login00 ~]$ workgroup -g it_css | |
| |
| [(it_css:traine)@login00 ~]$ echo $WORKDIR | |
| /work/it_css/users/traine | |
| |
| [(it_css:traine)@login00 ~]$ vpkg_info wrf | |
| [wrf] { | |
| contexts: all | |
| actions: { | |
| WRF_PREFIX=${VALET_PATH_PREFIX} (contexts: development) | |
| } | |
| https://www.mmm.ucar.edu/weather-research-and-forecasting-model | |
| A next-generation mesoscale numerical weather prediction system | |
| prefix: /opt/shared/wrf | |
| source file: /work/it_css/users/traine/sw/valet/wrf.vpkg_yaml | |
| default version: wrf/3.8.1:intel | |
| versions: { | |
| [wrf/3.8.1:intel] { | |
| contexts: all | |
| dependencies: { | |
| hdf5/1.8.20:openmpi,intel | |
| netcdf/4.1.3:intel | |
| } | |
| default configuration, Intel compilers and MPI | |
| prefix: /opt/shared/wrf/3.8.1-intel | |
| standard paths: { | |
| bin: /opt/shared/wrf/3.8.1-intel/bin | |
| } | |
| } | |
| } | |
| } | |
| </code> | |
| |
| To revert back to the defaults: | |
| |
| <code bash> | |
| [(it_css:traine)@login00 ~]$ exit | |
| exit | |
| |
| [traine@login00 ~]$ rm ~/.workdirs.yaml | |
| |
| traine@login00 ~]$ workgroup -g it_css | |
| |
| [(it_css:traine)@login00 ~]$ echo $WORKDIR | |
| /work/it_css | |
| |
| </code> | |