| Both sides previous revision Previous revision Next revision | Previous revision |
| abstract:darwin:app_dev:compute_env [2021-04-27 16:53] – [Using workgroup and directories] anita | abstract:darwin:app_dev:compute_env [2021-10-11 15:19] (current) – [Startup and logout scripts] anita |
|---|
| |
| ===== Environment variables ===== | ===== Environment variables ===== |
| |
| ==== Introduction ==== | |
| |
| Environment variables store dynamic system values that affect the user environment. For example, the ''PATH'' environment variable tells the operating system where to look for executables. Many UNIX commands and tools, such as the compilers, debuggers, profilers, editors, and applications with graphical user interfaces, often look at environment variables for information they need to function. The man pages for these programs typically have an ENVIRONMENT VARIABLES section with a list of variable names which tells how the program uses the values. | Environment variables store dynamic system values that affect the user environment. For example, the ''PATH'' environment variable tells the operating system where to look for executables. Many UNIX commands and tools, such as the compilers, debuggers, profilers, editors, and applications with graphical user interfaces, often look at environment variables for information they need to function. The man pages for these programs typically have an ENVIRONMENT VARIABLES section with a list of variable names which tells how the program uses the values. |
| All UNIX systems set up a default environment and provide users with the ability to execute additional UNIX commands to alter the environment. These commands are automatically sourced (executed) by your shell and define the normal and environmental variables, command aliases, and functions you need. Additionally, there is a final system-wide startup file that automatically makes global environment changes that IT sets for all users. | All UNIX systems set up a default environment and provide users with the ability to execute additional UNIX commands to alter the environment. These commands are automatically sourced (executed) by your shell and define the normal and environmental variables, command aliases, and functions you need. Additionally, there is a final system-wide startup file that automatically makes global environment changes that IT sets for all users. |
| |
| You can modify the default environment by adding lines __at the end__ of the ''~/.bash_profile'' file and the ''~/.bashrc'' file. These modifications affect shells started on the login node __and__ the compute nodes. | You can modify the default environment by adding lines __at the end__ of the ''~/.bash_profile'' file and the ''~/.bashrc'' file. These modifications affect shells started on the login node __and__ the compute nodes. In general we recommend that you __should not__ modify these files especially when software documentation refers to changing the ''PATH'' environment variable, instead use VALET to load the software. |
| |
| The ''~/.bash_profile'' file's commands are executed once at login. Add commands to this file to set your login environment and to run startup programs. | * The ''~/.bash_profile'' file's commands are executed once at login. Add commands to this file to set your login environment and to run startup programs. |
| | * The ''~/.bashrc'' file's commands are executed by each new shell you start (spawn). Add lines to this file to create aliases and bash functions. Commands such as **xterm** and **[[abstract:darwin:app_dev:compute_env#using-workgroup-and-directories|workgroup]]** automatically start a new shell and execute commands in the ''~/.bashrc'' file. The **salloc** command starts a shell on a compute node and will execute the ''~/.bashrc'' file from your home directory, but it does not execute the commands in the ''~/.bash_profile'' file. |
| |
| The ''~/.bashrc'' file's commands are executed by each new shell you start (spawn). Add lines to this file to create aliases and bash functions. Commands such as **xterm** and **[[abstract:darwin:app_dev:compute_env#using-workgroup-and-directories|workgroup]]** automatically start a new shell and execute commands in the ''~/.bashrc'' file. The **salloc** command starts a shell on a compute node and will execute the ''~/.bashrc'' file from your home directory, but it does not execute the | You may modify the IT-supplied ''~/.bash_udit'' file to be able to use several IT-supplied aliases (commands) and environment settings related to your [[abstract:darwin:app_dev:compute_env#using-workgroup-and-directories|workgroup]] and [[abstract:darwin:app_dev:compute_env#using-workgroup-and-directories|work directory]] . Edit ''.bash_udit'' and follow the directions in the file to activate these options. This is the ONLY way you should set your default workgroup at login. __DO NOT__ add the workgroup command to your ''.bashrc'' or ''.bash_profile'' as this will likely prevent you from logging in and will cause file transfer programs like WinSCP, sftp or Fetch to break. |
| commands in the ''~/.bash_profile'' file. | |
| | |
| You may modify the IT-supplied ''~/.bash_udit'' file to be able to use several IT-supplied aliases (commands) and environment settings related to your [[abstract:darwin:app_dev:compute_env#using-workgroup-and-directories|workgroup]] and [[abstract:darwin:app_dev:compute_env#using-workgroup-and-directories|work directory]] . Edit ''.bash_udit'' and follow the directions in the file to activate these options. | |
| |
| Exiting the login session or typing the **logout** command executes your ''~/.bash_logout'' file and terminates your session. Add commands to ''~/.bash_logout'' that you want to execute at logout. | Exiting the login session or typing the **logout** command executes your ''~/.bash_logout'' file and terminates your session. Add commands to ''~/.bash_logout'' that you want to execute at logout. |
| </code> | </code> |
| |
| <note tip>**Where to put startup commands:** You can put bash commands in either ''~/.bashrc'' or ''~/.bash_profile''. Here are general suggestions: | <note tip>**Where to put startup commands:** You can put bash commands in either ''~/.bashrc'' or ''~/.bash_profile''. Again we do not recommend modifying these files unless you really know what you are doing. Here are general suggestions: |
| * Even if you have favorite commands from other systems, start by using the supplied files and **only** modify ''.bash_udit'' for customization. | * Even if you have favorite commands from other systems, start by using the supplied files and **only** modify ''.bash_udit'' for customization. |
| * Add essential commands that you fully understand, and keep it simple. Quoting rules can be complicated. | * Add essential commands that you fully understand, and keep it simple. Quoting rules can be complicated. |
| * Be very careful of commands that may produce output. If you must, only execute them after a test to make sure there is a terminal to receive the output. Keep in mind using any commands that produce output may break other applications like file transfer (sftp, scp, WinSCP). | * Be very careful of commands that may produce output. If you must, only execute them after a test to make sure there is a terminal to receive the output. Keep in mind using any commands that produce output may break other applications like file transfer (sftp, scp, WinSCP). |
| * Do not include VALET commands as they produce output and will be a part of every job submitted which could cause conflicts with other applications you are trying to run in your job script. | * Do not include VALET commands as they produce output and will be a part of every job submitted which could cause conflicts with other applications you are trying to run in your job script. |
| * Keep a session open on the cluster, so when you make a change that prevents you from logging on you can reverse the last change, or copy the ''/etc/skel'' file to start over. | * Keep a session open on the cluster, so when you make a change that prevents you from logging on you can reverse the last change, or copy the original files from ''/opt/shared/templates/homedir/'' to start over. |
| </note> | </note> |
| |
| * set your workgroup (allocation group //allocation_group// name) with the **workgroup** command, | * set your workgroup (allocation group //allocation_group// name) with the **workgroup** command, |
| * change to your project directory with the **cd** command, | * change to your project directory with the **cd** command, |
| * connect to the compute node resources with **salloc** (or **sbatch**) command. | * connect to the compute node resources with **salloc** (or **sbatch**) command specifying a single partition your allocation workgroup has access based on resources requested for your allocation. |
| |
| ==== Connecting to login node ==== | === Connecting to login node === |
| |
| The system's initialization scripts set the values of some environment variables to help use the filesystems. | The system's initialization scripts set the values of some environment variables to help use the filesystems. |
| <note important>Clusters may be configured to have multiple login nodes, with one common name for connecting. For example, on the DARWIN cluster, the hostname may be set to ''login00'' or ''login01'', and the standard prompt and window title bar will indicate which login node on //darwin//. </note> | <note important>Clusters may be configured to have multiple login nodes, with one common name for connecting. For example, on the DARWIN cluster, the hostname may be set to ''login00'' or ''login01'', and the standard prompt and window title bar will indicate which login node on //darwin//. </note> |
| |
| ==== Setting workgroup ==== | === Setting workgroup === |
| |
| To use the compute node resources for a particular allocation group (workgroup), you need to use the **workgroup** command. | To use the compute node resources for a particular allocation group (workgroup), you need to use the **workgroup** command. |
| |
| ^ Variable ^ Example Value ^ Description ^ | ^ Variable ^ Example Value ^ Description ^ |
| | ''WORKDIR'' | ''/lustre/it_css'' | Allocation group directory, this is not writeable | | | ''WORKDIR'' | ''/lustre/it_css'' | Allocation workgroup directory, this is not writeable | |
| | ''WORKGROUP'' | ''it_css'' | Current allocation workgroup name | | | ''WORKGROUP'' | ''it_css'' | Current allocation workgroup name | |
| | ''WORKDIR_USER'' | ''/lustre/it_css/users/<<//uid//>>'' | Allocation group user directory | | | ''WORKDIR_USER'' | ''/lustre/it_css/users/<<//uid//>>'' | Allocation workgroup user directory | |
| | ''WORKDIR_SW'' | ''/lustre/it_css/sw'' | Allocation group software workgroup directory | | | ''WORKDIR_SW'' | ''/lustre/it_css/sw'' | Allocation workgroup software directory | |
| |
| <note tip>Use ''$WORKDIR'' when you refer to your allocation group directory. This will improve portability. </note> | <note tip>Use specific environment variables such as ''$WORKDIR_USERS'' when referring to your allocation workgroup user directory and ''$WORKDIR_SW'' when referring to your allocation workgroup software directory. This will improve portability. </note> |
| <note tip> | <note tip> |
| It is always important to be aware of your current allocation workgroup name. The standard prompt includes the workgroup name, added to your username and host. You must have a workgroup name in your prompt to use that allocation group's compute node resources to submit jobs using [[[[abstract/darwin/runjobs/schedule_jobs#batch-jobs-qsub|sbatch]] or salloc. An example prompt after the **workgroup** command, | It is always important to be aware of your current allocation workgroup name. The standard prompt includes the allocation workgroup name, added to your username and host. You must have an allocation workgroup name in your prompt to use that allocation group's compute node resources to [[abstract/darwin/runjobs/schedule_jobs|submit jobs]] using ''sbatch'' or ''salloc''. An example prompt after the **workgroup** command, |
| |
| ''[(it_css:traine)@login01.darwin ~]$ '' | ''[(it_css:traine)@login01.darwin ~]$ '' |
| === Changing directory === | === Changing directory === |
| |
| When you first connect to the login node, all your commands are executed from your home directory (''~''). Most of your work will be done in your allocation group work directory. The **workgroup** command has an option to start you in the allocation workgroup work directory. For example, | When you first connect to the login node, all your commands are executed from your home directory (''~''). Most of your work will be done in your allocation workgroup directory. The **workgroup** command has an option to start you in the allocation workgroup work directory. For example, |
| <code bash> | <code bash> |
| workgroup -cg it_css | workgroup -cg it_css |
| |
| <note important>Before submitting jobs you **must** first use the **workgroup** command. Type ''workgroup -h'' for additional information. | <note important>Before submitting jobs you **must** first use the **workgroup** command. Type ''workgroup -h'' for additional information. |
| | Both **sbatch** and **salloc** will start in the same project directory you set on the login node and will require a single [[abstract:darwin:runjobs:queues|partition]] to be specified to be able to submit a batch or interactive session. |
| Every time you connect to the login node, you should use **workgroup** to set the allocation workgroup you will being working in, and then **cd** to change to the directory for the project you will working on. Both **sbatch** and **salloc** will start in the same project directory you set on the login node. | |
| </note> | </note> |
| |