UDBUILD Software Deployment
The software installed and deployed on the Farber cluster each has it's own methods for compiling and installing. To manage this process, the HPC team has a set of standards and technology used to reduce complexity and bring consistency to the process.
Software is built, installed, and accessed using the
VALET system developed by Dr. Jeffrey Frey. IT
developed a set of software helper functions which can be access using VALET
by importing the udbuild
vpkg.
This page describes the filesystem layout used by IT, and the anatomy of the
udbuild
file used to build and deploy software. Throughout this process,
it is helpful to have an understanding of how to use
VALET to add and remove software packages from your
environment.
Filesystem
Software is deployed to /opt/shared
.
The udbuild
system defaults to this /opt/shared
location. However, this can be changed by setting the UDBUILD_HOME
environment variable before initializing the ubuild environment. A good
value for this environment variable for workgroup software is
$WORKDIR/sw
, and for personal software installation is
$HOME/sw
. Refer to workgroup software installs for help setting up your directories for workgroup storage.
Beneath this directory should be an attic
sub-directory for downloaded
software bundles, optionally an add-ons
directory for software with
optional add-ons, and one sub-directory for each package installed.
These sub-directories should always be in all lower-case letter. One more
layer down should be a directory for each version of the software installed.
It is important understand that on a complex cluster like Farber, the same
release of a software package may have multiple installations due to various
compiler and dependency package requirements. These directories are the
software installation roots.
Underneath the installation root should be a directory called src
, which
is the un-packed source bundle. Next to src
should be any bin
,
lib
, share
, etc. directories neccessary for the final deployment.
An illustrated example of the software directory structure is as such:
- opt
- shared
- atlas
- 3.10.2
- lib
- include
- src
udbuild
- build and install script for atlas 3.10.2
- 3.10.2-intel64
- attic
- python
- 2.7.8
- bin
- include
- lib
- share
- src
udbuild
- build and install script for python 2.7.8
- 3.2.5
- add-ons
- python2.7.8
- numpy
- 1.8.2
- bin
- lib
- src
udbuild
- build and install script for the numpy 1.8.2 installed for python 2.7.8
- ipython
- python3.2.5
- attic
Building
When building software, the base directory structure (including the attic
directory) should be created by you before proceeding further.
You should download the software source bundle into attic
. Then,
unpack the software bundle and rename the directory to src
as
above. This provides consistency in finding the source bundle and the
udbuild
file.
Examples of builds are provided below (after the udbuild function documentation).
udbuild functions
init_udbuildenv
This function initializes the udbuild environment. It ensures that you have
the required PKGNAME
and VERSION
environment variables defined, you do
not have VALET packages loaded before udbuild
in your VALET history (these
might affect your build), sets compiler variables like CC, FC, etc., then
finally sets your PREFIX
and VERSION
variables based on it's
command-line. These command-line options affect init_udbuildenv
:
- none - This is equivalent to not supplying any parameters
- python-addon - Ensure a python VALET package is loaded, and set PREFIX \
appropriately for that python version's add-on installation path
- r-addon - Ensure an R VALET package is loaded, and set PREFIX \
appropriately for that R version's add-on installation path
- Any other arguments are treated as the names of VALET packages which are \
loaded and added to the VERSION
environment variable.
After all of this, your PREFIX
variable will be set to
${UDBUILD_HOME:-/opt/shared}/$PKGNAME/$VERSION
valet
This function takes either the name of a package (e.g. openmpi
), or a
package name/version pair (e.g. openmpi/1.8.2
) and return true if there
is a VALET package loaded to satisfy this dependency, and false otherwise.
This function can be used along with any other shell constructs, such as
if
… else
… fi
, to modify the behaviour of a build.
version
This function takes a string and validates that it exists as a complete entry (i.e. starts, stops, or is bounded by hyphens) in the VERSION string.
This function can be used along with any other shell constructs, such as
if
… else
… fi
, to modify the behaviour of a build.
ifvalet
This function is shorthand for if valet "$1"; then shift; eval "$@"; fi
to make udbuild scripts simple to read and code.
ifversion
This function is shorthand for if version "$1"; then shift; eval "$@"; fi
to make udbuild scripts simple to read and code.
udbuildcapture
Put all screen output into a capture file. The main purpose of this is to log questions answered during an interactive isntall, to document what choices were made.
udbuildmon
This script is helpful to be run during the install phase of a build, for example:
udbuildmon make install
It will log all open
for write and mkdir
system calls and log them to
a file named udbuildmon.log
. You can use this log file to verify the
build did not write any files to unknown locations. This function should not
be necessary with cmake builds, as they normally
store this information in an install_manifest.txt
file.
udbuild script examples
simple
In this example, an easy-to-install software package called cmake is built
and isntalled. It has no software dependencies, and uses the standard
configure
, make
, make install
procedure used by very many
open source software packages.
To prepare for this build, you would want to create the following directories:
- udbuild
#/bin/bash -l PKGNAME=cmake #These are required variables and must VERSION=3.0.1 #be set before calling 'init_udbuildenv' vpkg_devrequire udbuild #Use VALET to load the udbuild environment init_udbuildenv #Initialize the udbuild environment ./configure --prefix=$PREFIX #Run your normal configure, without #having to define your own PREFIX #variable, because 'init_udbuildenv' did #that for you. make #normal make commands udbuildmon make install #wrap your 'make install' with the #'udbuildmon' function to log what files #and directories were changed.
It is imperitive to start udbuild scripts with the string #!/bin/bash -l
because this instructs bash to setup the VALET system.
medium
- udbuild
#!/bin/bash -l PKGNAME=cdo VERSION=1.6.4 vpkg_devrequire udbuild vpkg_devrequire netcdf/4.3.2 grib/1.12.3 udunits/2.2.17 proj/4 fftw/3 init_udbuildenv ./configure --prefix="$PREFIX" \ --with-szlib="$SZIP_PREFIX" \ --with-hdf5="$HDF5_PREFIX" \ --with-netcdf="$NETCDF_PREFIX" \ --with-grib_api="$GRIB_PREFIX" \ --with-proj="$PROJ_PREFIX" \ --with-udunits2="$UDUNITS_PREFIX" \ --with-threads=yes \ --with-curl=yes \ --with-libxml=yes make udbuildmon make install
In this example, we use vpkg_devrequire
to specify additional dependencies
needed to build the cdo
package. PREFIX
, however, will still be set
to /opt/shared/cdo/1.6.4
.
complex
- udbuild
#!/bin/bash -l PKGNAME=hdf4 VERSION=4.2.10 vpkg_devrequire udbuild szip/2.1 init_udbuildenv intel64 CFLAGS="$CFLAGS -fPIC" CONFIGURE_FLAGS='' ifvalet intel64 'CFLAGS="$CFLAGS -qopt-jump-tables=large"' ifversion sansnetcdf 'CONFIGURE_FLAGS="--disable-netcdf"' # Make shared libraries (sans fortran support): ./configure --prefix="$PREFIX" --enable-shared --disable-fortran \ --with-szlib=$SZIP_PREFIX $CONFIGURE_FLAGS udbuildmon make install make clean # Make fortran enabled HDF4: ./configure --prefix="$PREFIX" --disable-shared --enable-fortran \ --with-szlib=$SZIP_PREFIX $CONFIGURE_FLAGS udbuildmon make install
In this more complicated example, we still need dependencies, but this time
one of them will affect the PREFIX
variable. The Intel64 compiler will
be used, and PREFIX will be set to /opt/shared/hdf4/4.2.10-intel64
.
Furthermore, specific CFLAGS
changes will be made for this compiler.
This example also illustrates how the VERSION
string can be used. Here,
we would set additional flags for the ./configure
script if the VERSION
string were set to 4.2.10-sansnetcdf
. These options allow one build file
to build multiple versions of a package, and with only minor changes near
the top of the script (namely to the VERSION
variable and the
init_udbuildenv
command-line.
Another interesting thing we do here is to make sure the installation is
as complete as possible. HDF4 does not support shared object files for
fortran libraries. So, first we build the shared objects which are possible,
then we enable fortran and ensure the full compliment of archive .a
files
are present.
python
- udbuild
#!/bin/bash -l UDBUILD_HOME=$WORKDIR/sw PKGNAME=netcdf # Change this to the version of netcdf VERSION=1.1.7.1 # Change this to the version of python PY_VERS=2.7.8 # Load the udbuild package vpkg_devrequire udbuild # These are all the dependencies for netcdf specifying the # particular python version based on PY_VERS environment variable vpkg_devrequire python/$PY_VERS vpkg_devrequire python-numpy/python$PY_VERS vpkg_devrequire netcdf/4.3 # This initializes the environment for the build based on the # package name and version, and we use the install option # for a python add-on (python-addon) init_udbuildenv python-addon # These are environment variables needed for the install - hints for # what might be needed can be found in the setup.cfg.template file export HDF5_DIR=${HDF5_PREFIX} export NETCDF4_DIR=${NETCDF_PREFIX} export HDF4_DIR=${HDF4_PREFIX} # Standard install is 'python setup.py build' but we use $PYTHON to get # the correct version of python we are building the add-on for as defined # above rather than a default system version $PYTHON setup.py build $PYTHON setup.py install --prefix=$PREFIX
The python example above is used to install the netcdf library version 1.1.7.1 add-on for python version 2.7.8 in the shared research software directory $WORKDIR/sw
following the directory setup described for filesystem above. It is also considered a complex example since it displays the use of the option python-addon
for init_udbuildenv
to initialize the udbuild environment and ensure a python VALET package is loaded, sets PREFIX
appropriately for python version's add-on installation path, and utilizes $PYTHON
to get the correct version of python for the install. You can build and install the same version of netcdf 1.1.7.1 for a different version of python by simply changing PY_VERS
to the correct version and rerunning your udbuild script. To use the python add-on, you can setup a VALET package. Below is an example VALET 2.0 json package called python-netcdf
based on netcdf version 1.1.7.1 installed for both python versions 2.7.8 and 3.2.5. You need to change <WORKGROUP_NAME>
to match the corresponding shared research software directory workgroup name where you installed the python add-on software.
- python-netcdf.vpkg_json
# # This is the python-netcdf package for VALET 2.0 # supporting json package style. # { "python-netcdf": { # Change to your install location (i.e. <WORKGROUP_NAME>) "prefix": "/home/work/<WORKGROUP_NAME>/sw/python/add-ons", "url": "https://pypi.python.org/pypi/netCDF4", "description": "Python NetCDF", "default-version": "1.7.1.1-python2.7.8", "standard-paths": true, "development-env": false, "actions": [ { "action": "path-prepend", "variable": "PYTHONPATH", "value": "${VALET_PATH_PREFIX}/lib/python${PYTHON_VERSION_SHORT}/site-packages" } ], "versions": { "python2.7.8": { "alias-to": "1.1.7.1-python2.7.8" }, "python3.2.5": { "alias-to": "1.1.7.1-python3.2.5" }, "1.1.7.1-python2.7.8": { "description": "Version 1.1.7.1 and python 2.7.8", "prefix": "python2.7.8/netcdf/1.1.7.1", "dependencies": [ "python/2.7.8", "python-numpy/python2.7.8", "prefix": "python2.7.8/netcdf/1.1.7.1", "dependencies": [ "python/2.7.8", "python-numpy/python2.7.8", "netcdf/4.3" ] }, "1.1.7.1-python3.2.5": { "description": "Version 1.1.7.1 and python 3.2.5", "prefix": "python3.2.5/netcdf/1.1.7.1", "dependencies": [ "python/3.2.5", "python-numpy/python3.2.5", "netcdf/4.3" ] } } } }