===== UDBUILD Software Deployment =====
The software installed and deployed on the Farber cluster each has it's own
methods for compiling and installing.  To manage this process, the HPC team
has a set of standards and technology used to reduce complexity and bring
consistency to the process.

Software is built, installed, and accessed using the
[[software:valet:valet|VALET]] system developed by Dr. Jeffrey Frey.  IT
developed a set of software helper functions which can be access using VALET
by importing the ''udbuild'' vpkg.

This page describes the filesystem layout used by IT, and the anatomy of the
''udbuild'' file used to build and deploy software.  Throughout this process,
it is helpful to have an understanding of how to use
[[software:valet:valet|VALET]] to add and remove software packages from your
environment.

==== Filesystem ====
Software is deployed to ''/opt/shared''.
The ''udbuild'' system defaults to this ''/opt/shared''
location.  However, this can be changed by setting the ''UDBUILD_HOME''
environment variable before initializing the ubuild environment.  A good
value for this environment variable for workgroup software is
''$WORKDIR/sw'', and for personal software installation is
''$HOME/sw''. Refer to [[:technical:developer:workgroup-sw|workgroup software installs]] for help setting up your directories for workgroup storage.

Beneath this directory should be an ''attic'' sub-directory for downloaded
software bundles, optionally an ''add-ons'' directory for software with
optional add-ons, and one sub-directory for each package installed.
These sub-directories should always be in all lower-case letter.  One more
layer down should be a directory for each version of the software installed.
It is important understand that on a complex cluster like Farber, the same
release of a software package may have multiple installations due to various
compiler and dependency package requirements.  These directories are the
software installation roots.

Underneath the installation root should be a directory called ''src'', which
is the un-packed source bundle.  Next to ''src'' should be any ''bin'',
''lib'', ''share'', etc. directories neccessary for the final deployment.

An illustrated example of the software directory structure is as such:

  * opt
    * shared
      * atlas
         * 3.10.2
           * lib
           * include
           * src
             * ''udbuild''   <sup>// - build and install script for atlas 3.10.2//</sup>
         * 3.10.2-intel64
         * attic
      * python
         * 2.7.8
           * bin
           * include
           * lib
           * share
           * src
             * ''udbuild''   <sup>// - build and install script for python 2.7.8//</sup>
         * 3.2.5
         * add-ons
           * python2.7.8
             * numpy
               * 1.8.2
                 * bin
                 * lib
                 * src
                   * ''udbuild''  <sup>// - build and install script for the numpy 1.8.2 installed for python 2.7.8//</sup>
             * ipython
           * python3.2.5
         * attic


==== Building ====
When building software, the base directory structure (including the ''attic''
directory) should be created by you before proceeding further.
You should download the software source bundle into ''attic''.  Then,
unpack the software bundle and rename the directory to ''src'' as
above.  This provides consistency in finding the source bundle and the
''udbuild'' file.

Examples of builds are provided below (after the udbuild function
documentation).

=== udbuild functions ===

== init_udbuildenv ==
This function initializes the udbuild environment. It ensures that you have
the required ''PKGNAME'' and ''VERSION'' environment variables defined, you do
not have VALET packages loaded before ''udbuild'' in your VALET history (these
might affect your build), sets compiler variables like CC, FC, etc., then
finally sets your ''PREFIX'' and ''VERSION'' variables based on it's
command-line.  These command-line options affect ''init_udbuildenv'':

  * none - This is equivalent to not supplying any parameters
  * python-addon - Ensure a python VALET package is loaded, and set PREFIX \
    appropriately for that python version's add-on installation path
  * r-addon - Ensure an R VALET package is loaded, and set PREFIX \
    appropriately for that R version's add-on installation path
  * Any other arguments are treated as the names of VALET packages which are \
    loaded and added to the ''VERSION'' environment variable.

After all of this, your ''PREFIX'' variable will be set to

  ${UDBUILD_HOME:-/opt/shared}/$PKGNAME/$VERSION

== valet ==
This function takes either the name of a package (e.g. ''openmpi''), or a
package name/version pair (e.g. ''openmpi/1.8.2'') and return true if there
is a VALET package loaded to satisfy this dependency, and false otherwise.

This function can be used along with any other shell constructs, such as
''if'' ... ''else'' ... ''fi'', to modify the behaviour of a build.

== version ==
This function takes a string and validates that it exists as a complete
entry (i.e. starts, stops, or is bounded by hyphens) in the VERSION string.

This function can be used along with any other shell constructs, such as
''if'' ... ''else'' ... ''fi'', to modify the behaviour of a build.

== ifvalet ==
This function is shorthand for ''if valet "$1"; then shift; eval "$@"; fi''
to make udbuild scripts simple to read and code.

== ifversion ==
This function is shorthand for ''if version "$1"; then shift; eval "$@"; fi''
to make udbuild scripts simple to read and code.

== udbuildcapture ==
Put all screen output into a capture file.  The main purpose of this is to
log questions answered during an interactive isntall, to document what
choices were made.

== udbuildmon ==
This script is helpful to be run during the install phase of a build, for
example:

  udbuildmon make install

It will log all ''open'' for write and ''mkdir'' system calls and log them to
a file named ''udbuildmon.log''.  You can use this log file to verify the
build did not write any files to unknown locations.  This function should not
be necessary with [[http://www.cmake.org|cmake]] builds, as they normally
store this information in an ''install_manifest.txt'' file.

=== udbuild script examples ===

== simple ==
In this example, an easy-to-install software package called cmake is built
and isntalled.  It has no software dependencies, and uses the standard 
''configure'', ''make'', ''make install'' procedure used by very many
open source software packages.

To prepare for this build, you would want to create the following directories:


<file sh udbuild>
#/bin/bash -l

PKGNAME=cmake                       #These are required variables and must
VERSION=3.0.1                       #be set before calling 'init_udbuildenv'

vpkg_devrequire udbuild             #Use VALET to load the udbuild environment
init_udbuildenv                     #Initialize the udbuild environment

./configure --prefix=$PREFIX        #Run your normal configure, without
                                    #having to define your own PREFIX
                                    #variable, because 'init_udbuildenv' did
                                    #that for you.

make                                #normal make commands

udbuildmon make install             #wrap your 'make install' with the
                                    #'udbuildmon' function to log what files
                                    #and directories were changed.
</file>

It is imperitive to start udbuild scripts with the string ''#!/bin/bash -l''
because this instructs bash to setup the VALET system.

== medium ==
<file sh udbuild>
#!/bin/bash -l

PKGNAME=cdo
VERSION=1.6.4

vpkg_devrequire udbuild
vpkg_devrequire netcdf/4.3.2 grib/1.12.3 udunits/2.2.17 proj/4 fftw/3
init_udbuildenv

./configure --prefix="$PREFIX"                          \
                --with-szlib="$SZIP_PREFIX"             \
                --with-hdf5="$HDF5_PREFIX"              \
                --with-netcdf="$NETCDF_PREFIX"          \
                --with-grib_api="$GRIB_PREFIX"          \
                --with-proj="$PROJ_PREFIX"              \
                --with-udunits2="$UDUNITS_PREFIX"       \
                --with-threads=yes                      \
                --with-curl=yes                         \
                --with-libxml=yes

make

udbuildmon make install
</file>

In this example, we use ''vpkg_devrequire'' to specify additional dependencies
needed to build the ''cdo'' package.  ''PREFIX'', however, will still be set
to ''/opt/shared/cdo/1.6.4''.

== complex ==
<file sh udbuild>
#!/bin/bash -l

PKGNAME=hdf4
VERSION=4.2.10

vpkg_devrequire udbuild szip/2.1
init_udbuildenv intel64

CFLAGS="$CFLAGS -fPIC"
CONFIGURE_FLAGS=''

ifvalet intel64         'CFLAGS="$CFLAGS -qopt-jump-tables=large"'

ifversion sansnetcdf    'CONFIGURE_FLAGS="--disable-netcdf"'

# Make shared libraries (sans fortran support):
./configure --prefix="$PREFIX" --enable-shared --disable-fortran \
        --with-szlib=$SZIP_PREFIX $CONFIGURE_FLAGS

udbuildmon make install

make clean

# Make fortran enabled HDF4:

./configure --prefix="$PREFIX" --disable-shared --enable-fortran \
        --with-szlib=$SZIP_PREFIX $CONFIGURE_FLAGS

udbuildmon make install
</file>

In this more complicated example, we still need dependencies, but this time
one of them will affect the ''PREFIX'' variable.  The Intel64 compiler will
be used, and PREFIX will be set to ''/opt/shared/hdf4/4.2.10-intel64''.

Furthermore, specific ''CFLAGS'' changes will be made for this compiler.
This example also illustrates how the ''VERSION'' string can be used.  Here,
we would set additional flags for the ''./configure'' script if the ''VERSION''
string were set to ''4.2.10-sansnetcdf''.  These options allow one build file
to build multiple versions of a package, and with only minor changes near
the top of the script (namely to the ''VERSION'' variable and the
''init_udbuildenv'' command-line.

Another interesting thing we do here is to make sure the installation is
as complete as possible.  HDF4 does not support shared object files for
fortran libraries.  So, first we build the shared objects which are possible,
then we enable fortran and ensure the full compliment of archive ''.a'' files
are present.

== python ==
<file sh udbuild>
#!/bin/bash -l

UDBUILD_HOME=$WORKDIR/sw
PKGNAME=netcdf
# Change this to the version of netcdf
VERSION=1.1.7.1
# Change this to the version of python
PY_VERS=2.7.8

# Load the udbuild package
vpkg_devrequire udbuild

# These are all the dependencies for netcdf specifying the
# particular python version based on PY_VERS environment variable
vpkg_devrequire python/$PY_VERS
vpkg_devrequire python-numpy/python$PY_VERS
vpkg_devrequire netcdf/4.3

# This initializes the environment for the build based on the
# package name and version, and we use the install option
# for a python add-on (python-addon)
init_udbuildenv python-addon

# These are environment variables needed for the install - hints for
# what might be needed can be found in the setup.cfg.template file
export HDF5_DIR=${HDF5_PREFIX}
export NETCDF4_DIR=${NETCDF_PREFIX}
export HDF4_DIR=${HDF4_PREFIX}

# Standard install is 'python setup.py build' but we use $PYTHON to get
# the correct version of python we are building the add-on for as defined 
# above rather than a default system version
$PYTHON setup.py build

$PYTHON setup.py install --prefix=$PREFIX

</file>

The python example above is used to install the netcdf library version 1.1.7.1 add-on for python version 2.7.8 in the shared research software directory ''$WORKDIR/sw'' following the directory setup described for [[#filesystem]] above. It is also considered a complex example since it displays the use of the option ''python-addon'' for ''init_udbuildenv'' to initialize the udbuild environment and ensure a python VALET package is loaded, sets ''PREFIX'' appropriately for python version's add-on installation path, and utilizes ''$PYTHON'' to get the correct version of python for the install. You can build and install the same version of netcdf 1.1.7.1 for a different version of python by simply changing ''PY_VERS'' to the correct version and rerunning your udbuild script. To use the python add-on, you can setup a VALET package.  Below is an example VALET 2.0 json package called ''python-netcdf'' based on netcdf version 1.1.7.1 installed for both python versions 2.7.8 and 3.2.5. You need to change ''<WORKGROUP_NAME>'' to match the corresponding shared research software directory workgroup name where you installed the python add-on software.

<file sh python-netcdf.vpkg_json>
#
# This is the python-netcdf package for VALET 2.0
# supporting json package style.
#
{
  "python-netcdf": {
# Change to your install location (i.e. <WORKGROUP_NAME>)
    "prefix":           "/home/work/<WORKGROUP_NAME>/sw/python/add-ons",
    "url":              "https://pypi.python.org/pypi/netCDF4",
    "description":      "Python NetCDF",

    "default-version":  "1.7.1.1-python2.7.8",

    "standard-paths":   true,
    "development-env":  false,

    "actions": [
      {
        "action":       "path-prepend",
        "variable":     "PYTHONPATH",
        "value":        "${VALET_PATH_PREFIX}/lib/python${PYTHON_VERSION_SHORT}/site-packages"

      }
    ],

    "versions": {
      "python2.7.8": {
        "alias-to":     "1.1.7.1-python2.7.8"
      },
      "python3.2.5": {
        "alias-to":     "1.1.7.1-python3.2.5"
      },

      "1.1.7.1-python2.7.8": {
        "description":  "Version 1.1.7.1 and python 2.7.8",
        "prefix":       "python2.7.8/netcdf/1.1.7.1",
        "dependencies": [
                        "python/2.7.8",
                        "python-numpy/python2.7.8",
        "prefix":       "python2.7.8/netcdf/1.1.7.1",
        "dependencies": [
                        "python/2.7.8",
                        "python-numpy/python2.7.8",
                        "netcdf/4.3"
        ]
      },
      "1.1.7.1-python3.2.5": {
        "description":  "Version 1.1.7.1 and python 3.2.5",
        "prefix":       "python3.2.5/netcdf/1.1.7.1",
        "dependencies": [
                        "python/3.2.5",
                        "python-numpy/python3.2.5",
                        "netcdf/4.3"
        ]
      }
    }
  }
}
</file>