====== Computing Environment ======
===== UNIX shell =====

The //UNIX shell// is the interface to the UNIX operating system. The HPC cluster allows use of the enhanced Bourne shell //bash//, the enhanced C shell //tcsh//, and the enhanced Korn shell //zsh//. IT will primarily support //bash//, the default shell.

For most Linux systems, the //sh// shell is the //bash// shell and the //csh// shell is the //tcsh// shell. The remainder of this document will use only // bash// commands.

===== Environment variables =====

==== Introduction ====

Environment variables store dynamic system values that affect the user environment. For example, the ''PATH'' environment variable tells the operating system where to look for executables. Many UNIX commands and tools, such as the compilers, debuggers, profilers, editors, and applications with graphical user interfaces, often look at environment variables for information they need to function.  The man pages for these programs typically have an ENVIRONMENT VARIABLES section with a list of variable names which tells how the program uses the values.

This is why we encourage users to use [[abstract:caviness:app_dev:compute_env#using-valet-and-your-unix-environment|VALET]] to modify your environment versus explicitly setting environment variables.

In bash, a variable must be exported to be used as an environment variable. By convention, environment variables are all uppercase.  You can display a list of currently set environment variables by typing

<code bash>
    env
</code>

The **echo** and **export** commands will display and set environment variables.
^ Command ^ Results ^
|''  echo $''<<//varName//>> | Display specific environment variable |
|''  export ''<<//varName//>>''=''<<//varValue//>> | To set an environment variable to a value |
You can display specific environment variables by typing

For example,

<code bash >
    echo $HOME
    export FFLAGS='-g -Wall'
</code>

The variable ''FFLAGS'' will have the value '''-g -Wall''' in the shell and exported to programs run from this shell.
<note warning>Spaces are important. Do not put spaces around the equal sign. If the value has spaces, enclose the value in quotes.</note>
<note tip>If you see instructions that refer the ''setenv'' command, replace it with the ''export'' **bash** command.  Make sure you use equal signs, with no spaces.  The ''setenv'' **csh** command uses spaces instead of one equal.</note>
    
//This section uses the wiki's [[:#documentation-conventions|documentation conventions]].//
==== Startup and logout scripts ====

All UNIX systems set up a default environment and provide users with the ability to execute additional UNIX commands to alter the environment. These commands are automatically sourced (executed) by your shell and define the normal and environmental variables, command aliases, and functions you need. Additionally, there is a final system-wide startup file that automatically makes global environment changes that IT sets for all users.

You can modify the default environment by adding lines __at the end__ of the ''~/.bash_profile'' file and the ''~/.bashrc'' file. These modifications affect shells started on the login node __and__ the compute nodes. In general we recommend that you __should not__ modify these files especially when software documentation refers to changing the ''PATH'' environment variable, instead use VALET to load the software.

  * The ''~/.bash_profile'' file's commands are executed once at login. Add commands to this file to set your login environment and to run startup programs.
  * The ''~/.bashrc'' file's commands are executed by each new shell you start (spawn). Add lines to this file to create aliases and bash functions. Commands such as **xterm** and **[[abstract:caviness:app_dev:compute_env#using-workgroup-and-directories|workgroup]]** automatically start a new shell and execute commands in the ''~/.bashrc'' file.  The **salloc** command starts a shell on a compute node and will execute the ''~/.bashrc'' file from your home directory, but it does not execute the commands in the ''~/.bash_profile'' file. 

You may modify the IT-supplied ''~/.bash_udit'' file to be able to use several IT-supplied aliases (commands) and environment settings related to your [[abstract:caviness:app_dev:compute_env#using-workgroup-and-directories|workgroup]] and [[abstract:caviness:app_dev:compute_env#using-workgroup-and-directories|work directory]] . Edit ''.bash_udit'' and follow the directions in the file to activate these options. This is the ONLY way you should set your default workgroup at login.  __DO NOT__ add the workgroup command to your ''.bashrc'' or ''.bash_profile'' as this will likely prevent you from logging in and will cause file transfer programs like WinSCP, sftp or Fetch to break.

Exiting the login session or typing the **logout** command executes your ''~/.bash_logout'' file and terminates your session. Add commands to ''~/.bash_logout'' that you want to execute at logout.

To restore the ''.bash_profile'', ''.bashrc'', ''.bash_udit'' and ''.bash_logout'' files in your home directory to their original state, type:

<code bash>
    cp /opt/shared/templates/homedir/.bash* $HOME
</code>
<note tip>**Where to put startup commands:** You can put bash commands in either ''~/.bashrc'' or ''~/.bash_profile''. Again we do not recommend modifying these files unless you really know what you are doing. Here are general suggestions:
  * Even if you have favorite commands from other systems, start by using the supplied files and **only** modify ''.bash_udit'' for customization.
  * Add essential commands that you fully understand, and keep it simple.  Quoting rules can be complicated.
  * Do not depend on the order of command execution.  Do not assume your environment, set in ''.bash_profile'', will be available when the commands in ''.bashrc'' are executed.
  * Do not include commands that spawn new shells, such as ''workgroup''.
  * Be very careful of commands that may produce output.  If you must, only execute them after a test to make sure there is a terminal to receive the output.  Keep in mind using any commands that produce output may break other applications like file transfer (sftp, scp, WinSCP).
  * Do not include VALET commands as they produce output and will be a part of every job submitted which could cause conflicts with other applications you are trying to run in your job script.
  * Keep a session open on the cluster, so when you make a change that prevents you from logging on you can reverse the last change, or copy the original files from  ''/opt/shared/templates/homedir/'' to start over.
</note>

==== Using workgroup and directories ====

There are some key environment variables that are set for you, and are important for your work on any cluster. They are used to find directories for your projects.  These environment variables are set on initial connection to a cluster, and will be changed if you
  * set your workgroup (cluster group //investing-entity// name) with the **workgroup** command,
  * change to your project directory with the **cd** command,
  * connect to the compute node with **salloc** (or **sbatch**) command.

=== Connecting to login node ===

The system's initialization scripts set the values of some environment variables to help use the filesystems. 

^ Variable        ^ Value                              ^ Description  ^
| ''HOSTNAME''        | <<//hostname//>>                   | Host name  |
| ''USER''            | <<//login_name//>>                 | Login name  |
| ''HOME''            | ''/home/''<<//uid//>>              | Your home directory  |


<note tip>
The initialization scripts also set the //standard prompt// with your login name and a shortened host name.
For example, if your <<//hostname//>> is ''caviness.hpc.udel.edu'' and your <<//login_name//>> is ''traine'', then the //standard prompt// will be

''[traine@login01 ~]$ ''
</note>
<note important>Clusters may be configured to have multiple login nodes, with one common name for connecting.  For example, on the Caviness cluster, the hostname may be set to ''login00'' or ''login01'', but the standard prompt and window title bar are set to //caviness//. </note>
=== Setting workgroup ===

To use the compute nodes for a particular //investing-entity// cluster group (workgroup), you need to use the **workgroup** command. 

For example,
<code bash>
    workgroup –g it_css
</code>

starts a new shell for the workgroup ''it_css'', and sets the environment variables:

^ Variable        ^ Example Value                ^ Description  ^
| ''WORKDIR''         | ''/work/it_css''    | Group private work directory |                         
| ''WORKGROUP''       | ''it_css''                   | Current workgroup name |

<note tip>Use ''$WORKDIR'' when you refer to your group private directory.  This will improve portability. </note>
<note tip>
It is always important to be aware of your current workgroup name.  The standard prompt includes the workgroup name, added to your username and host.  You must have a workgroup name in your prompt to use that cluster group's (//investing-entity//) compute node resources to submit jobs using [[[[abstract/caviness/runjobs/schedule_jobs#batch-jobs-qsub|sbatch]] or salloc. An example prompt after the **workgroup** command,

''[(it_css:traine)@login01 ~]$ ''
</note>

=== Changing directory ===

When you first connect to the login node, all your commands are executed from your home directory.  Most of your work will be done in your private group work directory.  The **workgroup** command has an option to start you in the private group work directory. For example,
<code bash>
    workgroup -cg it_css
</code>
will spawn a new shell in the workgroup directory for ''it_css''.

You can always use the ''cd'' bash command. 

For example,
<code bash>
    cd project/fuelcell
    cd /lustre/scratch/traine/project/fuelcell
    cd $WORKDIR/project/fuelcell
</code>
The first is using a path name relative to the current working directory (implied ''./'').  The second to use the full path (''$WOKDIR'' always begins with a ''/'')
In all cases the directory is changed, and the ''$PWD'' environment variable is set:

^ Variable    ^ Example Value                           ^ Description  ^
| ''PWD''         | ''/lustre/scratch/traine/project/fuelcell''    | Print (current) working directory |

<note tip>
It is always important to be aware of your current working directory.  The standard prompt ends with the basename of ''PWD''.  In these two examples the basename is the same, ''traine'', but the standard bash ''PROMPT_COMMAND'', which is executed every time you change directories, will put the full path of your current working directory in your window title. For example,

''traine@caviness:/work/it_css/users/traine''
</note>

=== Connecting to a compute node ===
To run a job on the compute nodes, you must submit your job script using ''sbatch'' or start an interactive session using ''salloc''.  In both cases, you will be connected to one of your investing-entity's compute nodes with a clean environment.  Do not rely on the environment you set on the login node. The variables ''USER'', ''HOME'', ''WORKGROUP'', ''WORKDIR'', and ''PWD'' are all set on the compute node to match the ones you had on the login node, but two variables are set to node-specific values:

^ Variable        ^ Example Value    ^ Description  ^
| ''HOSTNAME''    | ''r00n17''       | Compute node name  |
| ''TMPDIR''      | ''/tmp/1185/0''  | Temporary disk space |

An empty directory is created by the SLURM job scheduler that is associated with your job and defined as ''TMPDIR''. This is a safe place to store temporary files that will not interfere with other jobs and tasks you or other members of your group may be executing.  This directory is automatically emptied on normal termination of your job. This way the usage on the node scratch filesystem will not grow over time.

<note important>Before submitting jobs you **must** first use the **workgroup** command. Type ''workgroup -h'' for additional information.

Every time you connect to the login node, you should use **workgroup** to set the group you will being working in, and then **cd** to change to the directory for the project you will working on. Both **sbatch** and **salloc** will start in the same project directory you set on the login node.
</note>

//This section uses the wiki's [[:#documentation-conventions|documentation conventions]].//
===== Using VALET and your UNIX environment =====

The UD-developed //VALET// system facilitates your use of compilers, libraries, programming tools and application software. It provides a uniform mechanism for setting up a package's required UNIX environment. VALET is a recursive acronym for **//V//**//ALET //**//A//**//utomates //**//L//**//inux //**//E//**//nvironment //**//T//**//asks//. It provides functionality similar to the **Modules** package used at other HPC sites.

VALET commands set the basic environment for software. This may include the ''PATH'', ''MANPATH'', ''INFOPATH'', ''LDPATH'', ''LIBPATH'' and ''LD_LIBRARY_PATH'' environment variables, compiler flags, software directory locations, and license paths. This reduces the need for you to set them or update them yourself when changes are made to system and application software. For example, you might find several versions for a single package name, such as Mathematica/8 and Mathematica/8.0.4. You can even apply VALET commands to packages that you install or alter its actions by customizing VALET's configuration files. Type **man valet** for instructions or see the [[:software:valet:valet|VALET software documentation]] for complete details.

The table below shows the basic informational commands for VALET. In subsequent sections, VALET commands are illustrated in the contexts of application development (e.g., compiling, using libraries) and running IT-installed applications.

^ Command        ^ Function     ^
| ''vpkg_help''  | VALET help.  |
| ''vpkg_list''  | List the packages that have VALET configuration files.  |
| ''vpkg_versions ''<<//pkgid//>>  | List versions available for a single package.  |
| ''vpkg_info ''<<//pkgid//>>  | Show information for a single package (or package version).  |
| ''vpkg_require ''<<//pkgid//>>  | Configure environment for one or more VALET packages.  |
| ''vpkg_devrequire ''<<//pkgid//>> | Configure environment for one or more VALET packages including software development variables such as ''CPPFLAGS'' and ''LDFLAGS''.  |
| ''vpkg_rollback ''<<//# or all//>> | Each time VALET changes the environment, it makes a snapshot of your environment to which it can return.\\ ''vpkg_rollback'' attempts to restore the UNIX environment to its previous state. You can specify a number (''#'') to revert one or more prior changes to the environment or ''all'' to remove all changes.  |
| ''vpkg_history''  | List the versioned packages that have been added to the environment.  |
| ''man valet''  | Complete documentation of VALET commands.  |

==== Having a personal Work directory apart from the common workgroup directory ====

Users can maintain their own work directories (''$WORKDIR'') apart from the common work directory of the workgroup. It is also possible to make VALET software to look into this user level work directory. Users can now add ''~/.workdirs.yaml'' to override what gets set for ''$WORKDIR'' for workgroups to which they are members. Detailed explanation follows.

<code bash>
[traine@login00 ~]$ workdir -g it_css
/work/it_css

[traine@login00 ~]$ echo "it_css: /work/it_css/users/traine" > ~/.workdirs.yaml

[traine@login00 ~]$ workdir -g it_css
/work/it_css/users/traine
</code>

Now inside the new work directory, users can create sw/valet directory to add valet packages

<code bash>
[traine@login00 ~]$ ls -l /work/it_css/users/traine/sw/valet
total 11
-rw-r--r-- 1 frey it_nss 480 Sep 21 16:06 wrf.vpkg_yaml
</code>

From this point, valet looks at this directory for possible valet packages and also the ''$WORKDIR'' after changing to workgroup will point to the path overridden in ''workdirs.yaml'' file.

<code bash>
traine@login00 ~]$ workgroup -g it_css

[(it_css:traine)@login00 ~]$ echo $WORKDIR
/work/it_css/users/traine

[(it_css:traine)@login00 ~]$ vpkg_info wrf
[wrf] {
  contexts: all
  actions: {
    WRF_PREFIX=${VALET_PATH_PREFIX} (contexts: development)
  }
  https://www.mmm.ucar.edu/weather-research-and-forecasting-model
  A next-generation mesoscale numerical weather prediction system
  prefix: /opt/shared/wrf
  source file: /work/it_css/users/traine/sw/valet/wrf.vpkg_yaml
  default version: wrf/3.8.1:intel
  versions: {
    [wrf/3.8.1:intel] {
      contexts: all
      dependencies: {
        hdf5/1.8.20:openmpi,intel
        netcdf/4.1.3:intel
      }
      default configuration, Intel compilers and MPI
      prefix: /opt/shared/wrf/3.8.1-intel
      standard paths: {
        bin: /opt/shared/wrf/3.8.1-intel/bin
      }
    }
  }
}
</code>

To revert back to the defaults:

<code bash>
[(it_css:traine)@login00 ~]$ exit
exit

[traine@login00 ~]$ rm ~/.workdirs.yaml 

traine@login00 ~]$ workgroup -g it_css

[(it_css:traine)@login00 ~]$ echo $WORKDIR
/work/it_css

</code>