The software installed and deployed on the Farber cluster each has it's own methods for compiling and installing. To manage this process, the HPC team has a set of standards and technology used to reduce complexity and bring consistency to the process.

Software is built, installed, and accessed using the VALET system developed by Dr. Jeffrey Frey. IT developed a set of software helper functions which can be access using VALET by importing the udbuild vpkg.

This page describes the filesystem layout used by IT, and the anatomy of the udbuild file used to build and deploy software. Throughout this process, it is helpful to have an understanding of how to use VALET to add and remove software packages from your environment.

Software is deployed to /opt/shared. The udbuild system defaults to this /opt/shared location. However, this can be changed by setting the UDBUILD_HOME environment variable before initializing the ubuild environment. A good value for this environment variable for workgroup software is $WORKDIR/sw, and for personal software installation is $HOME/sw. Refer to workgroup software installs for help setting up your directories for workgroup storage.

Beneath this directory should be an attic sub-directory for downloaded software bundles, optionally an add-ons directory for software with optional add-ons, and one sub-directory for each package installed. These sub-directories should always be in all lower-case letter. One more layer down should be a directory for each version of the software installed. It is important understand that on a complex cluster like Farber, the same release of a software package may have multiple installations due to various compiler and dependency package requirements. These directories are the software installation roots.

Underneath the installation root should be a directory called src, which is the un-packed source bundle. Next to src should be any bin, lib, share, etc. directories neccessary for the final deployment.

An illustrated example of the software directory structure is as such:

• opt
  • shared
    • atlas
      • 3.10.2
        • lib
        • include
        • src
          • udbuild ^{- build and install script for atlas 3.10.2}
      • 3.10.2-intel64
      • attic
    • python
      • 2.7.8
        • bin
        • include
        • lib
        • share
        • src
          • udbuild ^{- build and install script for python 2.7.8}
      • 3.2.5
      • add-ons
        • python2.7.8
          • numpy
            • 1.8.2
              • bin
              • lib
              • src
                • udbuild ^{- build and install script for the numpy 1.8.2 installed for python 2.7.8}
          • ipython
        • python3.2.5
      • attic

When building software, the base directory structure (including the attic directory) should be created by you before proceeding further. You should download the software source bundle into attic. Then, unpack the software bundle and rename the directory to src as above. This provides consistency in finding the source bundle and the udbuild file.

Examples of builds are provided below (after the udbuild function documentation).

This function initializes the udbuild environment. It ensures that you have the required PKGNAME and VERSION environment variables defined, you do not have VALET packages loaded before udbuild in your VALET history (these might affect your build), sets compiler variables like CC, FC, etc., then finally sets your PREFIX and VERSION variables based on it's command-line. These command-line options affect init_udbuildenv:

• none - This is equivalent to not supplying any parameters
• python-addon - Ensure a python VALET package is loaded, and set PREFIX \

appropriately for that python version's add-on installation path

• r-addon - Ensure an R VALET package is loaded, and set PREFIX \

appropriately for that R version's add-on installation path

• Any other arguments are treated as the names of VALET packages which are \

loaded and added to the VERSION environment variable.

After all of this, your PREFIX variable will be set to

${UDBUILD_HOME:-/opt/shared}/$PKGNAME/$VERSION

This function takes either the name of a package (e.g. openmpi), or a package name/version pair (e.g. openmpi/1.8.2) and return true if there is a VALET package loaded to satisfy this dependency, and false otherwise.

This function can be used along with any other shell constructs, such as if … else … fi, to modify the behaviour of a build.

This function takes a string and validates that it exists as a complete entry (i.e. starts, stops, or is bounded by hyphens) in the VERSION string.

This function can be used along with any other shell constructs, such as if … else … fi, to modify the behaviour of a build.

This function is shorthand for if valet "$1"; then shift; eval "$@"; fi to make udbuild scripts simple to read and code.

This function is shorthand for if version "$1"; then shift; eval "$@"; fi to make udbuild scripts simple to read and code.

Put all screen output into a capture file. The main purpose of this is to log questions answered during an interactive isntall, to document what choices were made.

This script is helpful to be run during the install phase of a build, for example:

udbuildmon make install

It will log all open for write and mkdir system calls and log them to a file named udbuildmon.log. You can use this log file to verify the build did not write any files to unknown locations. This function should not be necessary with cmake builds, as they normally store this information in an install_manifest.txt file.

In this example, an easy-to-install software package called cmake is built and isntalled. It has no software dependencies, and uses the standard configure, make, make install procedure used by very many open source software packages.

To prepare for this build, you would want to create the following directories:

udbuild

#/bin/bash -l
 
PKGNAME=cmake                       #These are required variables and must
VERSION=3.0.1                       #be set before calling 'init_udbuildenv'
 
vpkg_devrequire udbuild             #Use VALET to load the udbuild environment
init_udbuildenv                     #Initialize the udbuild environment
 
./configure --prefix=$PREFIX        #Run your normal configure, without
                                    #having to define your own PREFIX
                                    #variable, because 'init_udbuildenv' did
                                    #that for you.
 
make                                #normal make commands
 
udbuildmon make install             #wrap your 'make install' with the
                                    #'udbuildmon' function to log what files
                                    #and directories were changed.

It is imperitive to start udbuild scripts with the string #!/bin/bash -l because this instructs bash to setup the VALET system.

udbuild

#!/bin/bash -l
 
PKGNAME=cdo
VERSION=1.6.4
 
vpkg_devrequire udbuild
vpkg_devrequire netcdf/4.3.2 grib/1.12.3 udunits/2.2.17 proj/4 fftw/3
init_udbuildenv
 
./configure --prefix="$PREFIX"                          \
                --with-szlib="$SZIP_PREFIX"             \
                --with-hdf5="$HDF5_PREFIX"              \
                --with-netcdf="$NETCDF_PREFIX"          \
                --with-grib_api="$GRIB_PREFIX"          \
                --with-proj="$PROJ_PREFIX"              \
                --with-udunits2="$UDUNITS_PREFIX"       \
                --with-threads=yes                      \
                --with-curl=yes                         \
                --with-libxml=yes
 
make
 
udbuildmon make install

In this example, we use vpkg_devrequire to specify additional dependencies needed to build the cdo package. PREFIX, however, will still be set to /opt/shared/cdo/1.6.4.

udbuild

#!/bin/bash -l
 
PKGNAME=hdf4
VERSION=4.2.10
 
vpkg_devrequire udbuild szip/2.1
init_udbuildenv intel64
 
CFLAGS="$CFLAGS -fPIC"
CONFIGURE_FLAGS=''
 
ifvalet intel64         'CFLAGS="$CFLAGS -qopt-jump-tables=large"'
 
ifversion sansnetcdf    'CONFIGURE_FLAGS="--disable-netcdf"'
 
# Make shared libraries (sans fortran support):
./configure --prefix="$PREFIX" --enable-shared --disable-fortran \
        --with-szlib=$SZIP_PREFIX $CONFIGURE_FLAGS
 
udbuildmon make install
 
make clean
 
# Make fortran enabled HDF4:
 
./configure --prefix="$PREFIX" --disable-shared --enable-fortran \
        --with-szlib=$SZIP_PREFIX $CONFIGURE_FLAGS
 
udbuildmon make install

In this more complicated example, we still need dependencies, but this time one of them will affect the PREFIX variable. The Intel64 compiler will be used, and PREFIX will be set to /opt/shared/hdf4/4.2.10-intel64.

Furthermore, specific CFLAGS changes will be made for this compiler. This example also illustrates how the VERSION string can be used. Here, we would set additional flags for the ./configure script if the VERSION string were set to 4.2.10-sansnetcdf. These options allow one build file to build multiple versions of a package, and with only minor changes near the top of the script (namely to the VERSION variable and the init_udbuildenv command-line.

Another interesting thing we do here is to make sure the installation is as complete as possible. HDF4 does not support shared object files for fortran libraries. So, first we build the shared objects which are possible, then we enable fortran and ensure the full compliment of archive .a files are present.

udbuild

#!/bin/bash -l
 
UDBUILD_HOME=$WORKDIR/sw
PKGNAME=netcdf
# Change this to the version of netcdf
VERSION=1.1.7.1
# Change this to the version of python
PY_VERS=2.7.8
 
# Load the udbuild package
vpkg_devrequire udbuild
 
# These are all the dependencies for netcdf specifying the
# particular python version based on PY_VERS environment variable
vpkg_devrequire python/$PY_VERS
vpkg_devrequire python-numpy/python$PY_VERS
vpkg_devrequire netcdf/4.3
 
# This initializes the environment for the build based on the
# package name and version, and we use the install option
# for a python add-on (python-addon)
init_udbuildenv python-addon
 
# These are environment variables needed for the install - hints for
# what might be needed can be found in the setup.cfg.template file
export HDF5_DIR=${HDF5_PREFIX}
export NETCDF4_DIR=${NETCDF_PREFIX}
export HDF4_DIR=${HDF4_PREFIX}
 
# Standard install is 'python setup.py build' but we use $PYTHON to get
# the correct version of python we are building the add-on for as defined 
# above rather than a default system version
$PYTHON setup.py build
 
$PYTHON setup.py install --prefix=$PREFIX

The python example above is used to install the netcdf library version 1.1.7.1 add-on for python version 2.7.8 in the shared research software directory $WORKDIR/sw following the directory setup described for filesystem above. It is also considered a complex example since it displays the use of the option python-addon for init_udbuildenv to initialize the udbuild environment and ensure a python VALET package is loaded, sets PREFIX appropriately for python version's add-on installation path, and utilizes $PYTHON to get the correct version of python for the install. You can build and install the same version of netcdf 1.1.7.1 for a different version of python by simply changing PY_VERS to the correct version and rerunning your udbuild script. To use the python add-on, you can setup a VALET package. Below is an example VALET 2.0 json package called python-netcdf based on netcdf version 1.1.7.1 installed for both python versions 2.7.8 and 3.2.5. You need to change <WORKGROUP_NAME> to match the corresponding shared research software directory workgroup name where you installed the python add-on software.

python-netcdf.vpkg_json

#
# This is the python-netcdf package for VALET 2.0
# supporting json package style.
#
{
  "python-netcdf": {
# Change to your install location (i.e. <WORKGROUP_NAME>)
    "prefix":           "/home/work/<WORKGROUP_NAME>/sw/python/add-ons",
    "url":              "https://pypi.python.org/pypi/netCDF4",
    "description":      "Python NetCDF",
 
    "default-version":  "1.7.1.1-python2.7.8",
 
    "standard-paths":   true,
    "development-env":  false,
 
    "actions": [
      {
        "action":       "path-prepend",
        "variable":     "PYTHONPATH",
        "value":        "${VALET_PATH_PREFIX}/lib/python${PYTHON_VERSION_SHORT}/site-packages"
 
      }
    ],
 
    "versions": {
      "python2.7.8": {
        "alias-to":     "1.1.7.1-python2.7.8"
      },
      "python3.2.5": {
        "alias-to":     "1.1.7.1-python3.2.5"
      },
 
      "1.1.7.1-python2.7.8": {
        "description":  "Version 1.1.7.1 and python 2.7.8",
        "prefix":       "python2.7.8/netcdf/1.1.7.1",
        "dependencies": [
                        "python/2.7.8",
                        "python-numpy/python2.7.8",
        "prefix":       "python2.7.8/netcdf/1.1.7.1",
        "dependencies": [
                        "python/2.7.8",
                        "python-numpy/python2.7.8",
                        "netcdf/4.3"
        ]
      },
      "1.1.7.1-python3.2.5": {
        "description":  "Version 1.1.7.1 and python 3.2.5",
        "prefix":       "python3.2.5/netcdf/1.1.7.1",
        "dependencies": [
                        "python/3.2.5",
                        "python-numpy/python3.2.5",
                        "netcdf/4.3"
        ]
      }
    }
  }
}