Table of Contents

Computing Environment

UNIX shell

The UNIX shell is the interface to the UNIX operating system. The HPC cluster allows use of the enhanced Bourne shell bash, the enhanced C shell tcsh, and the enhanced Korn shell zsh. IT will primarily support bash, the default shell.

For most Linux systems, the sh shell is the bash shell and the csh shell is the tcsh shell. The remainder of this document will use only bash commands.

The following sections use the wiki's documentation conventions.

Environment variables

Introduction

Environment variables store dynamic system values that affect the user environment. For example, the PATH environment variable tells the operating system where to look for executables. Many UNIX commands and tools, such as the compilers, debuggers, profilers, editors, and applications with graphical user interfaces, often look at environment variables for information they need to function. The man pages for these programs typically have an ENVIRONMENT VARIABLES section with a list of variable names which tells how the program uses the values.

In bash, a variable must be exported to be used as an environment variable. By convention, environment variables are all uppercase. You can display a list of currently set environment variables by typing 

    env

The echo and export commands will display and set environment variables.

Command Results
echo $«varName» Display specific environment variable
export «varName»=«varValue» To set an environment variable to a value

You can display specific environment variables by typing

For example, 

    echo $HOME
    export FFLAGS='-g -Wall'

The variable FFLAGS will have the value '-g -Wall' in the shell and exported to programs run from this shell.

Spaces are important. Do not put spaces around the equal sign. If the value has spaces, enclose the value in quotes.
If you see instructions that refer the setenv command, replace it with the export bash command. Make sure you use equal signs, with no spaces. The setenv csh command uses spaces instead of one equal.

Startup and logout scripts

All UNIX systems set up a default environment and provide users with the ability to execute additional UNIX commands to alter the environment. These commands are automatically sourced (executed) by your shell and define the normal and environmental variables, command aliases, and functions you need. Additionally, there is a final system-wide startup file that automatically makes global environment changes that IT sets for all users.

You can modify the default environment by adding lines at the end of the ~/.bash_profile file and the ~/.bashrc file. These modifications affect shells started on the login node and the compute nodes.

The ~/.bash_profile file's commands are executed once at login. Add commands to this file to set your login environment and to run startup programs.

The ~/.bashrc file's commands are executed by each new shell you start (spawn). Add lines to this file to create aliases and bash functions. Commands such as xterm and workgroup automatically start a new shell and execute commands in the ~/.bashrc file. The qlogin command starts a shell on a compute node and will execute the ~/.bashrc file from your home directory, but it does not execute the commands in the ~/.bash_profile file.

You may modify the IT-supplied ~/.bash_udit file to be able to use several IT-supplied aliases (commands) and environment settings related to your workgroup and work directory . Edit .bash_udit and follow the directions in the file to activate these options.

Exiting the login session or typing the logout command executes your ~/.bash_logout file and terminates your session. Add commands to ~/.bash_logout that you want to execute at logout.

To restore the .bash_profile, .bashrc, .bash_udit and .bash_logout files in your home directory to their original state, type: 

    cp /opt/shared/templates/homedir/.bash* $HOME
Where to put startup commands: You can put bash commands in either ~/.bashrc or ~/.bash_profile. Here are general suggestions:
  • Even if you have favorite commands from other systems, start by using the supplied files. Only modify .bash_udit for customization.
  • Add essential commands that you fully understand, and keep it simple. Quoting rules can be complicated.
  • Do not depend on the order of command execution. Do not assume your environment, set in .bash_profile, will be available when the commands in .bashrc are executed.
  • Do not include commands that spawn new shells, such as workgroup.
  • Be very careful of commands that may produce output. If you must, only execute them after a test to make sure there is a terminal to receive the output. Keep in mind using any commands that produce output may break other applications like file transfer (sftp, scp, WINscp).
  • Keep a session open on the cluster, so when you make a change that prevents you from logging on you can reverse the last change, or copy the /etc/skel file to start over.

Using workgroup and directories

There are some key environment variables that are set for you, and are important for your work on any cluster. They are used to find directories for your projects. These environment variables are set on initial connection to a cluster, and will be changed if you

Connecting to login node

The system's initialization scripts set the values of some environment variables to help use the filesystems.

Variable Value Description
HOSTNAME «hostname» Host name
USER «login_name» Login name
HOME /home/«uid» Your home directory
The initialization scripts also set the standard prompt with your login name and a shortened host name. For example, if your «hostname» is farber.hpc.udel.edu and your «login_name» is traine, then the standard prompt will be

[traine@farber ~]$

Clusters may be configured to have multiple login nodes, with one common name for connecting. For example, on the Farber cluster, the hostname may be set to login000 or login001, but the standard prompt and window title bar are set to farber.

Setting workgroup

To use the compute nodes for a particular investing-entity cluster group (workgroup), you need to use the workgroup command.

For example, 

    workgroup –g it_css

starts a new shell for the workgroup it_css, and sets the environment variables:

Variable Example Value Description
WORKDIR /home/work/it_css Group private work directory
WORKGROUP it_css Current workgroup name
Use $WORKDIR when you refer to your group private directory. This will improve portability.
It is always important to be aware of your current workgroup name. The standard prompt includes the workgroup name, added to your username and host. You must have a workgroup name in your prompt to use that cluster group's (investing-entity) compute nodes to submit jobs using qsub or qlogin. An example prompt after the workgroup command,

[(it_css:traine)@farber ~]$

Changing directory

When you first connect to the login node, all your commands are executed from your home directory. Most of your work will be done in your private group work directory. The workgroup command has an option to start you in the private group work directory. For example, 

    workgroup -cg it_css

will spawn a new shell in the workgroup directory for it_css.

You can always use the cd bash command.

For example, 

    cd project/fuelcell
    cd /lustre/scratch/traine/project/fuelcell
    cd $WORKDIR/project/fuelcell

The first is using a path name relative to the current working directory (implied ./). The second to use the full path ($WOKDIR always begins with a /) In all cases the directory is changed, and the $PWD environment variable is set:

Variable Example Value Description
PWD /lustre/scratch/traine/project/fuelcell Print (current) working directory
It is always important to be aware of your current working directory. The standard prompt ends with the basename of PWD. In these two examples the basename is the same, traine, but the standard bash PROMPT_COMMAND, which is executed every time you change directories, will put the full path of your current working directory in your window title. For example,

traine@farber:/home/work/it_css/users/traine

Connecting to a compute node

To run a job on the compute nodes, you must submit your job script using qsub or start an interactive session using qlogin. In both cases, you will be connected to one of your investing-entity's compute nodes with a clean environment. Do not rely on the environment you set on the login node. The variables USER, HOME, WORKGROUP, WORKDIR, and PWD are all set on the compute node to match the ones you had on the login node, but two variables are set to node-specific values:

Variable Example Value Description
HOSTNAME n017 Compute node name
TMPDIR /scratch/67209.1.it_css-qrsh.q Temporary disk space

An empty directory is created by the Grid Engine scheduler that is associated with your job and defined as TMPDIR. This is a safe place to store temporary files that will not interfere with other jobs and tasks you or other members of your group may be executing. This directory is automatically emptied on normal termination of your job. This way the usage on the node scratch filesystem will not grow over time.

Before submitting jobs you must first use the workgroup command. Type workgroup -h for additional information.

Every time you connect to the login node, you should use workgroup to set the group you will being working in, and then cd to change to the directory for the project you will working on. Both qsub and qlogin will start in the same project directory you set on the login node.

This section uses the wiki's documentation conventions.

Using VALET and your UNIX environment

The UD-developed VALET system facilitates your use of compilers, libraries, programming tools and application software. It provides a uniform mechanism for setting up a package's required UNIX environment. VALET is a recursive acronym for VALET Automates Linux Environment Tasks. It provides functionality similar to the Modules package used at other HPC sites.

VALET commands set the basic environment for software. This may include the PATH, MANPATH, INFOPATH, LDPATH, LIBPATH and LD_LIBRARY_PATH environment variables, compiler flags, software directory locations, and license paths. This reduces the need for you to set them or update them yourself when changes are made to system and application software. For example, you might find several versions for a single package name, such as Mathematica/8 and Mathematica/8.0.4. You can even apply VALET commands to packages that you install or alter its actions by customizing VALET's configuration files. Type man valet for instructions or see the VALET software documentation for complete details.

The table below shows the basic informational commands for VALET. In subsequent sections, VALET commands are illustrated in the contexts of application development (e.g., compiling, using libraries) and running IT-installed applications.

Command Function
vpkg_help VALET help.
vpkg_list List the packages that have VALET configuration files.
vpkg_versions «pkgid» List versions available for a single package.
vpkg_info «pkgid» Show information for a single package (or package version).
vpkg_rollback all Each time VALET changes the environment, it makes a snapshot of your environment to which it can return.
vpkg_rollback attempts to restore the UNIX environment to its previous state. You can specify a number in place of all to limit the rollback.
vpkg_history List the versioned packages that have been added to the environment.
man valet Complete documentation of VALET commands.