Table of Contents

UDBUILD Software Deployment

The software installed and deployed on the Caviness cluster each has it's own methods for compiling and installing. To manage this process, the HPC team has a set of standards and technology used to reduce complexity and bring consistency to the process.

Software is built, installed, and accessed using the VALET system developed by Dr. Jeffrey Frey. IT developed a set of software helper functions which can be access using VALET by importing the udbuild vpkg.

This page describes the filesystem layout used by IT, and the anatomy of the udbuild file used to build and deploy software. Throughout this process, it is helpful to have an understanding of how to use VALET to add and remove software packages from your environment.

Filesystem

Software is deployed to /opt/shared. The udbuild system defaults to this /opt/shared location. However, this can be changed by setting the UDBUILD_HOME environment variable before initializing the ubuild environment. A good value for this environment variable for workgroup software is $WORKDIR/sw, and for personal software installation is $HOME/sw. Refer to workgroup software installs for help setting up your directories for workgroup storage.

Beneath this directory should be an attic sub-directory for downloaded software bundles, optionally an add-ons directory for software with optional add-ons, and one sub-directory for each package installed. These sub-directories should always be in all lower-case letter. One more layer down should be a directory for each version of the software installed. It is important understand that on a complex cluster like Caviness, the same release of a software package may have multiple installations due to various compiler and dependency package requirements. These directories are the software installation roots.

Underneath the installation root should be a directory called src, which is the un-packed source bundle. Next to src should be any bin, lib, share, etc. directories neccessary for the final deployment.

An illustrated example of the software directory structure is as such:

Building

When building software, the base directory structure (including the attic directory) should be created by you before proceeding further. You should download the software source bundle into attic. Then, unpack the software bundle and rename the directory to src as above. This provides consistency in finding the source bundle and the udbuild file.

Examples of builds are provided below (after the udbuild function documentation).

udbuild functions

init_udbuildenv

This function initializes the udbuild environment. It ensures that you have the required PKGNAME and VERSION environment variables defined, you do not have VALET packages loaded before udbuild in your VALET history (these might affect your build), sets compiler variables like CC, FC, etc., then finally sets your PREFIX and VERSION variables based on it's command-line. These command-line options affect init_udbuildenv:

appropriately for that python version's add-on installation path

appropriately for that R version's add-on installation path

appropriately for that Node.JS versions' add-on installation path

loaded and added to the VERSION environment variable.

After all of this, your PREFIX variable will be set to 

${UDBUILD_HOME:-/opt/shared}/$PKGNAME/$VERSION
debug

Drop into a debug shell. If the debug shell is exited cleanly, then the udbuild script will continue from there. This is a useful routine to use when creating a udbuild script. You may want to build the script based on documentation, and run a debug function between a configure and make step to verify the environment looks sane. After the first successful compile, you can then remove the debug line.

download

The download function takes a mandatory and optional argument. The mandatory, first, argument is the URL to download. The resulting file will be named after the last part of the URL unless the optional second argument is specified. In this case, the resulting file will be named after the second argument.

If a file with the same name already exists, the download exits successfully without doing anything. If you wish to re-download the archive, delete or rename the existing one.

unpack

The unpack function takes a mandatory and optional argument. The mandatory, first, argument is the name of an archive file (tar.gz, tar.bz2, tar.xz, zip, etc.) to extract. The archive will be unpacked into a directory named src under the install prefix (versioned directory for installation) unless the optional second argument is specified, then it will be used in place of the name src. This directory, and its parents if necessary, will be created prior to extraction. Source archives using the tar format customarily have a single top-level directory entry which contains the package name and version, this is automatically stripped from the extracted archive. After completing the extraction process, the unpack function places the udbuild script into the newly created directory to prepare the script for configure and make steps.

If the src (or alternately specified) directory exists, then the archive is not extracted over it. In this case, the function returns successfully without doing anything. If you wish to force a new extraction, remove or rename the existing src directory.

create_valet_template

Create a YAML based valet package file template and place it in the attic directory if one can be found, and the same directory as the udbuild script if one cannot. This template is helpful, but usually cannot just be copied into place. For example, it only knows about the version of the software it is installing, and copying the file blindly would remove entries for all other versions. Furthermore, the "dependencies" entry is filled with all loaded valet packages, even if they are dependencies of dependencies, and not needed to be explicitly listed.

valet

This function takes either the name of a package (e.g. openmpi), or a package name/version pair (e.g. openmpi/1.8.2) and return true if there is a VALET package loaded to satisfy this dependency, and false otherwise.

This function can be used along with any other shell constructs, such as if … else … fi, to modify the behaviour of a build.

version

This function takes a string and validates that it exists as a complete entry (i.e. starts, stops, or is bounded by hyphens) in the VERSION string.

This function can be used along with any other shell constructs, such as if … else … fi, to modify the behavior of a build.

package

This function takes a string and validates that it exists as part of the final package name, which may include features. This is useful for matching features which are specified to configure a software build but don't show up in the version string or require a valet package be available, the string "threads" for OpenMP is a good example of using this feature.

ifvalet

This function is shorthand for if valet "$1"; then shift; eval "$@"; fi to make udbuild scripts simple to read and code.

ifversion

This function is shorthand for if version "$1"; then shift; eval "$@"; fi to make udbuild scripts simple to read and code.

ifpackage

This function is shorthand for if package "$1"; then shift; eval "$@"; fi to make udbuild scripts simple to read and code.

udbuildcapture

Put all screen output into a capture file. The main purpose of this is to log questions answered during an interactive isntall, to document what choices were made.

udbuildmon

This script is helpful to be run during the install phase of a build, for example: 

udbuildmon make install

It will log all open for write and mkdir system calls and log them to a file named udbuildmon.log. You can use this log file to verify the build did not write any files to unknown locations. This function should not be necessary with cmake builds, as they normally store this information in an install_manifest.txt file.

apath

Append a path to a variable and ensure that variable is exported. The required first argument is the environment variable name, all remaining arguments are paths to append to the end of the environment variable. A colon (:) character is used as the delimiter, as is standard in path environment variables.

ppath

Prepend a path to a variable similar to apath, but instead of adding the path to the end, add it to the beginning. Arguments are the same as apath

rpath

Remove a path from an environment variable. The required first argument is the environment variable name. All remaining arguments are removed from the environment variable. If an entry exists multiple times, all instances are removed.

aflag

Append a flag to an environment variable. The required first argument is the environment variable name. All remaining arguments are added to the environment variable. The aflag variable works under two contexts, which depend on the status of the first argument. If it is an already defined bash array variable type, then the remaining arguments are added as new elements in the array. In all other cases, the remaining arguments are added to the string using a space character as a delimiter.

Using a bash array has the advantage of allowing flags which contain whitespace characters. If this is a requirement, the following steps should be undertaken: 

CONFIG=()                            #Declare CONFIG as a bash array
aflag CONFIG --prefix="$PREFIX"      #Specify flags, this one is regular
aflag CONFIG --title="My Software"   #Specify flags, his one with spaces
./configure  "${CONFIG[@]}"          #bash syntax for arrays as arguments
pflag

Prepend flags to an environment variable. This is the same as the aflag function, but it puts its arguments at the beginning of the variable. Its arguments are identical.

rflag

Remove a flag from an environment variable. This works similar to the rpath variable and also supports bash arrays.

udexpect

This is a wrapper around the TCL expect utility to simplify the process of answering questions for interactive builds. This function accepts an expect script as STDIN (the normal method is via HERE-DOC) and provides all the basics of running expect. Some standard responses are provided to simplify the process:

  1. enter - Send a carrige return as if the user pressed their "Enter" key
  2. yes - Send the string "yes" as if the user typed "yes" and pressed "Enter"
  3. no - Send "no" and press "Enter"
  4. y - Send "y" and press "Enter"
  5. n - Send "n" and press "Enter"
  6. respond text - Send text and press "Enter"
  7. keypress c - Send the character c and DO NOT press "Enter"
  8. user - Prompt the person at the keyboard for a respone, and send it, press "Enter"
makeflags_set

Update a file (presumably a Makefile and specified as the first argument) which uses the syntax "key=value" and update the value of the second argument to be that of the third argument. This is a simple helper function to make it simple to edit basic information in a Makefile.

makeflags_prepend

Update a file similar to makeflags_set, except prepend the third argument to the existing value, instead of replacing it.

makeflags_append

Update a file similar to makeflags_set, except append the third argument to the existing value, instead of replacing it.

udbuild script examples

simple

In this example, an easy-to-install software package called cmake is built and isntalled. It has no software dependencies, and uses the standard configure, make, make install procedure used by very many open source software packages.

To prepare for this build, you would want to create the following directories:

udbuild
#/bin/bash -l
 
PKGNAME=cmake                       #These are required variables and must
VERSION=3.11.3                      #be set before calling 'init_udbuildenv'
                                    #Setting the "SITEURL" is not required, but is
                                    #helpful later.
SITEURL=https://cmake.org/files/v${VERSION%.*}
 
PKGINFO='Cross-Platform Make'       #Helpful for the ''create_valet_template'' function,
URLINFO=http://www.cmake.org        #but not required at all
 
vpkg_devrequire udbuild/2           #Use VALET to load the udbuild v2 environment
init_udbuildenv                     #Initialize the udbuild environment
 
                                    #Download the source file if it doesn't already exist
download $SITEURL/$PKGNAME-$VERSION.tar.gz
unpack $PKGNAME-$VERSION.tar.gz     #Unpack the source file into $PREFIX/src and cd there
 
create_valet_template               #Create a template file for valet in the attic (you
                                    #can't just copy this into place)
 
./configure --prefix=$PREFIX        #Run your normal configure, without
                                    #having to define your own PREFIX
                                    #variable, because 'init_udbuildenv' did
                                    #that for you.
 
make                                #normal make commands
 
udbuildmon make install             #wrap your 'make install' with the
                                    #'udbuildmon' function to log what files
                                    #and directories were changed.

It is imperitive to start udbuild scripts with the string #!/bin/bash -l because this instructs bash to setup the VALET system.

medium
udbuild
#!/bin/bash -l
 
PKGNAME=cdo
VERSION=1.9.4
SITEURL=https://code.mpimet.mpg.de/attachments/download/17374/
 
vpkg_devrequire udbuild/2 eccodes/2.8.0:threads proj/5.1.0
vpkg_devrequire netcdf/4.6.1 udunits/2.2.26 fftw/3.3.8
init_udbuildenv
 
create_valet_template
 
download $SITEURL/$PKGNAME-$VERSION.tar.gz
unpack $PKGNAME-$VERSION.tar.gz
 
aflag CONFIG --with-szlib="$SZIP_PREFIX"                
aflag CONFIG --with-hdf5="$HDF5_PREFIX"                 
aflag CONFIG --with-netcdf="$NETCDF_PREFIX"             
aflag CONFIG --with-eccodes="$ECCODES_PREFIX"   
aflag CONFIG --with-proj="$PROJ_PREFIX"                 
aflag CONFIG --with-udunits2="$UDUNITS_PREFIX"  
aflag CONFIG --with-threads=yes                         
aflag CONFIG --with-curl=yes                    
aflag CONFIG --with-libxml=yes
 
./configure --prefix="$PREFIX" $CONFIG
 
sed -i 's/#include <string>/#include <string.h>/' src/modules.cc
 
make
 
udbuildmon make install

In this example, we use vpkg_devrequire to specify additional dependencies needed to build the cdo package. PREFIX, however, will still be set to /opt/shared/cdo/1.6.4.

complex
udbuild
#!/bin/bash -l
 
PKGNAME=hdf4
VERSION=4.2.13
SITEURL=https://support.hdfgroup.org/ftp/HDF/HDF_Current/src
 
PKGINFO='HDF4: Data Model, Library, & File Format'
URLINFO=http://www.hdfgroup.org/products/hdf4/
 
vpkg_devrequire udbuild szip/2.1.1
init_udbuildenv
 
create_valet_template
 
download $SITEURL/hdf-$VERSION.tar.bz2
unpack hdf-$VERSION.tar.bz2
 
aflag CFLAGS -fPIC                           
if valet intel; then   
        aflag CFLAGS -qopt-jump-tables=large
fi             
 
aflag CONFIG --disable-netcdf
aflag CONFIG --with-szlib=$SZIP_PREFIX
 
# Make shared libraries (sans fortran support):
./configure --prefix="$PREFIX" --enable-shared --disable-fortran $CONFIG
 
make install
 
make clean
 
# Make fortran enabled HDF4:
./configure --prefix="$PREFIX" --disable-shared --enable-fortran $CONFIG
 
make install

In this more complicated example, we still need dependencies, but this time one of them will affect the PREFIX variable. The Intel64 compiler will be used, and PREFIX will be set to /opt/shared/hdf4/4.2.10-intel64.

Furthermore, specific CFLAGS changes will be made for this compiler. This example also illustrates how the VERSION string can be used. Here, we would set additional flags for the ./configure script if the VERSION string were set to 4.2.10-sansnetcdf. These options allow one build file to build multiple versions of a package, and with only minor changes near the top of the script (namely to the VERSION variable and the init_udbuildenv command-line.

Another interesting thing we do here is to make sure the installation is as complete as possible. HDF4 does not support shared object files for fortran libraries. So, first we build the shared objects which are possible, then we enable fortran and ensure the full compliment of archive .a files are present.

python
udbuild-cdf
#!/bin/bash -l
 
PKGNAME=cdf
VERSION=20180613
PY_VER=3.6.5
NETCDF_VER=4.6.1
 
PKGINFO='Python NetCDF, HDF5, HDF4, and dependent Modules'
URLINFO=https://pypi.org/
 
vpkg_devrequire udbuild python/$PY_VER netcdf/$NETCDF_VER
init_udbuildenv python-addon
 
create_valet_template
 
ppath PATH        "$PREFIX/bin"
apath LD_RUN_PATH "$PREFIX/lib"
 
pip_install CDF cdflib netCDF4 h5netCDF wera2netcdf Puppy nco
pip_install python-hdf4 hdf5able hdf5pickle hdf5storage
pip_install dtt2hdf ascii2hdf5 h5json h5pyd hdf5_matlab_reader
pip_install HDFconvert mrr2c hdfdict LazyHDF5 simpletraj mdtraj
pip_install mriqc ncagg
pip_install 'MDAnalysis[analysis, AMBER]'

The python example above is used to install the cdf and related modules as an add-on for python version 3.6.5. It is also considered a complex example since it displays the use of the option python-addon for init_udbuildenv to initialize the udbuild environment and ensure a python VALET package is loaded, sets PREFIX appropriately for python version's add-on installation path, and utilizes the udbuild pip_install function to call pip with using the correct PREFIX. To use the python add-on module bundle, you can setup a VALET package. Below is an example VALET yaml package created from the create_valet_template function. It should be trimmed, as szip, hdf4, and hdf5 are all dependencies of netcdf. The dependencies section should look like the vpkg_devrequire calls of the udbuild script.

python-cdf.vpkg_yaml
python-cdf:
  description:          Python NetCDF, HDF5, HDF4, and dependent Modules
  url:                  https://pypi.org/
  prefix:               /opt/shared/python/add-ons
 
  default-version:      "3.6.5:20180613"
 
  versions:
    "3.6.5:20180613":
      description:      cdf compiled with system compilers
      prefix:           python3.6.5/cdf/20180613
      dependencies:
        -               python/3.6.5
        -               szip/2.1.1
        -               hdf4/4.2.13
        -               hdf5/1.10.2
        -               netcdf/4.6.1