===== UDBUILD Software Deployment =====
The software installed and deployed on the Caviness cluster each has it's own
methods for compiling and installing.  To manage this process, the HPC team
has a set of standards and technology used to reduce complexity and bring
consistency to the process.

Software is built, installed, and accessed using the
[[software:valet:valet|VALET]] system developed by Dr. Jeffrey Frey.  IT
developed a set of software helper functions which can be access using VALET
by importing the ''udbuild'' vpkg.

This page describes the filesystem layout used by IT, and the anatomy of the
''udbuild'' file used to build and deploy software.  Throughout this process,
it is helpful to have an understanding of how to use
[[software:valet:valet|VALET]] to add and remove software packages from your
environment.

==== Filesystem ====
Software is deployed to ''/opt/shared''.
The ''udbuild'' system defaults to this ''/opt/shared''
location.  However, this can be changed by setting the ''UDBUILD_HOME''
environment variable before initializing the ubuild environment.  A good
value for this environment variable for workgroup software is
''$WORKDIR/sw'', and for personal software installation is
''$HOME/sw''. Refer to [[:abstract:caviness:.install_software:workgroup-sw|workgroup software installs]] for help setting up your directories for workgroup storage.

Beneath this directory should be an ''attic'' sub-directory for downloaded
software bundles, optionally an ''add-ons'' directory for software with
optional add-ons, and one sub-directory for each package installed.
These sub-directories should always be in all lower-case letter.  One more
layer down should be a directory for each version of the software installed.
It is important understand that on a complex cluster like Caviness, the same
release of a software package may have multiple installations due to various
compiler and dependency package requirements.  These directories are the
software installation roots.

Underneath the installation root should be a directory called ''src'', which
is the un-packed source bundle.  Next to ''src'' should be any ''bin'',
''lib'', ''share'', etc. directories neccessary for the final deployment.

An illustrated example of the software directory structure is as such:

  * opt
    * shared
      * atlas
         * 3.10.3
         * 3.10.3-intel
         * attic
           * ''udbuild''  <sup>// - build and install script for atlas//</sup>
      * python
         * 2.7.8
         * 3.2.5
         * add-ons
           * python2.7.15
             * mpi
               * 20180613
           * python3.2.5
         * attic
           * ''udbuild''   <sup>// - build and install script for python//</sup>

==== Building ====
When building software, the base directory structure (including the ''attic''
directory) should be created by you before proceeding further.
You should download the software source bundle into ''attic''.  Then,
unpack the software bundle and rename the directory to ''src'' as
above.  This provides consistency in finding the source bundle and the
''udbuild'' file.

Examples of builds are provided below (after the udbuild function
documentation).

=== udbuild functions ===

== init_udbuildenv ==
This function initializes the udbuild environment. It ensures that you have
the required ''PKGNAME'' and ''VERSION'' environment variables defined, you do
not have VALET packages loaded before ''udbuild'' in your VALET history (these
might affect your build), sets compiler variables like CC, FC, etc., then
finally sets your ''PREFIX'' and ''VERSION'' variables based on it's
command-line.  These command-line options affect ''init_udbuildenv'':

  * none - This is equivalent to not supplying any parameters
  * python-addon - Ensure a python VALET package is loaded, and set PREFIX \
    appropriately for that python version's add-on installation path
  * r-addon - Ensure an R VALET package is loaded, and set PREFIX \
    appropriately for that R version's add-on installation path
  * node-addon - Ensure a Node.JS version is loaded and set the PREFIX \
    appropriately for that Node.JS versions' add-on installation path
  * Any other arguments are treated as the names of VALET packages which are \
    loaded and added to the ''VERSION'' environment variable.

After all of this, your ''PREFIX'' variable will be set to

  ${UDBUILD_HOME:-/opt/shared}/$PKGNAME/$VERSION

== debug ==
Drop into a debug shell.  If the debug shell is exited cleanly, then the
udbuild script will continue from there.  This is a useful routine to use
when creating a udbuild script.  You may want to build the script based
on documentation, and run a ''debug'' function between a configure and
make step to verify the environment looks sane.  After the first successful
compile, you can then remove the debug line.

== download ==
The ''download'' function takes a mandatory and optional argument.  The mandatory,
first, argument is the URL to download.  The resulting file will be named after
the last part of the URL unless the optional second argument is specified.  In this
case, the resulting file will be named after the second argument.

If a file with the same name already exists, the download exits successfully without
doing anything.  If you wish to re-download the archive, delete or rename the existing
one.

== unpack ==
The ''unpack'' function takes a mandatory and optional argument.  The mandatory,
first, argument is the name of an archive file (tar.gz, tar.bz2, tar.xz, zip, etc.)
to extract.  The archive will be unpacked into a directory named ''src'' under the
install prefix (versioned directory for installation) unless the optional second argument
is specified, then it will be used in place of the name ''src''.  This directory, and
its parents if necessary, will be created prior to extraction.  Source archives using
the ''tar'' format customarily have a single top-level directory entry which contains
the package name and version, this is automatically stripped from the extracted archive.
After completing the extraction process, the ''unpack'' function places the udbuild
script into the newly created directory to prepare the script for configure and make
steps.

If the ''src'' (or alternately specified) directory exists, then the archive is not
extracted over it.  In this case, the function returns successfully without doing
anything.  If you wish to force a new extraction, remove or rename the existing
''src'' directory.

== create_valet_template ==
Create a YAML based valet package file template and place it in the ''attic'' directory
if one can be found, and the same directory as the udbuild script if one cannot.  This
template is helpful, but usually cannot just be copied into place.  For example, it only
knows about the version of the software it is installing, and copying the file blindly would
remove entries for all other versions.  Furthermore, the "dependencies" entry is filled with
all loaded valet packages, even if they are dependencies of dependencies, and not needed
to be explicitly listed.

== valet ==
This function takes either the name of a package (e.g. ''openmpi''), or a
package name/version pair (e.g. ''openmpi/1.8.2'') and return true if there
is a VALET package loaded to satisfy this dependency, and false otherwise.

This function can be used along with any other shell constructs, such as
''if'' ... ''else'' ... ''fi'', to modify the behaviour of a build.

== version ==
This function takes a string and validates that it exists as a complete
entry (i.e. starts, stops, or is bounded by hyphens) in the VERSION string.

This function can be used along with any other shell constructs, such as
''if'' ... ''else'' ... ''fi'', to modify the behavior of a build.

== package ==
This function takes a string and validates that it exists as part of
the final package name, which may include features.  This is useful for
matching features which are specified to configure a software build but
don't show up in the version string or require a valet package be
available, the string "threads" for OpenMP is a good example of using
this feature.

== ifvalet ==
This function is shorthand for ''if valet "$1"; then shift; eval "$@"; fi''
to make udbuild scripts simple to read and code.

== ifversion ==
This function is shorthand for ''if version "$1"; then shift; eval "$@"; fi''
to make udbuild scripts simple to read and code.

== ifpackage ==
This function is shorthand for ''if package "$1"; then shift; eval "$@"; fi''
to make udbuild scripts simple to read and code.

== udbuildcapture ==
Put all screen output into a capture file.  The main purpose of this is to
log questions answered during an interactive isntall, to document what
choices were made.

== udbuildmon ==
This script is helpful to be run during the install phase of a build, for
example:

  udbuildmon make install

It will log all ''open'' for write and ''mkdir'' system calls and log them to
a file named ''udbuildmon.log''.  You can use this log file to verify the
build did not write any files to unknown locations.  This function should not
be necessary with [[http://www.cmake.org|cmake]] builds, as they normally
store this information in an ''install_manifest.txt'' file.

== apath ==
Append a path to a variable and ensure that variable is exported.  The required
first argument is the environment variable name, all remaining arguments are
paths to append to the end of the environment variable.  A colon (:) character
is used as the delimiter, as is standard in path environment variables.

== ppath ==
Prepend a path to a variable similar to ''apath'', but instead of adding the
path to the end, add it to the beginning.  Arguments are the same as ''apath''

== rpath ==
Remove a path from an environment variable.  The required first argument is the
environment variable name.  All remaining arguments are removed from the
environment variable.  If an entry exists multiple times, all instances are
removed.

== aflag ==
Append a flag to an environment variable.  The required first argument is the
environment variable name.  All remaining arguments are added to the environment
variable.  The ''aflag'' variable works under two contexts, which depend on the
status of the first argument.  If it is an already defined bash array variable
type, then the remaining arguments are added as new elements in the array.  In
all other cases, the remaining arguments are added to the string using a space
character as a delimiter.

Using a bash array has the advantage of allowing flags which contain whitespace
characters.  If this is a requirement, the following steps should be undertaken:

  CONFIG=()                            #Declare CONFIG as a bash array
  aflag CONFIG --prefix="$PREFIX"      #Specify flags, this one is regular
  aflag CONFIG --title="My Software"   #Specify flags, his one with spaces
  ./configure  "${CONFIG[@]}"          #bash syntax for arrays as arguments

== pflag ==
Prepend flags to an environment variable.  This is the same as the ''aflag''
function, but it puts its arguments at the beginning of the variable.  Its
arguments are identical.

== rflag ==
Remove a flag from an environment variable.  This works similar to the
''rpath'' variable and also supports bash arrays.

== udexpect ==
This is a wrapper around the TCL ''expect'' utility to simplify the process of
answering questions for interactive builds.  This function accepts an expect
script as STDIN (the normal method is via HERE-DOC) and provides all the basics
of running ''expect''.  Some standard responses are provided to simplify the
process:

  - enter - Send a carrige return as if the user pressed their "Enter" key
  - yes - Send the string "yes" as if the user typed "yes" and pressed "Enter"
  - no - Send "no" and press "Enter"
  - y - Send "y" and press "Enter"
  - n - Send "n" and press "Enter"
  - respond text - Send //text// and press "Enter"
  - keypress c - Send the character //c// and DO NOT press "Enter"
  - user - Prompt the person at the keyboard for a respone, and send it, press "Enter"

== makeflags_set ==
Update a file (presumably a Makefile and specified as the first argument) which uses
the syntax "key=value" and update the value of the second argument to be that of the
third argument.  This is a simple helper function to make it simple to edit basic
information in a Makefile.

== makeflags_prepend ==
Update a file similar to ''makeflags_set'', except prepend the third argument to the
existing value, instead of replacing it.

== makeflags_append ==
Update a file similar to ''makeflags_set'', except append the third argument to the
existing value, instead of replacing it.

=== udbuild script examples ===

== simple ==
In this example, an easy-to-install software package called cmake is built
and isntalled.  It has no software dependencies, and uses the standard 
''configure'', ''make'', ''make install'' procedure used by very many
open source software packages.

To prepare for this build, you would want to create the following directories:


<file sh udbuild>
#/bin/bash -l

PKGNAME=cmake                       #These are required variables and must
VERSION=3.11.3                      #be set before calling 'init_udbuildenv'
                                    #Setting the "SITEURL" is not required, but is
                                    #helpful later.
SITEURL=https://cmake.org/files/v${VERSION%.*}

PKGINFO='Cross-Platform Make'       #Helpful for the ''create_valet_template'' function,
URLINFO=http://www.cmake.org        #but not required at all

vpkg_devrequire udbuild/2           #Use VALET to load the udbuild v2 environment
init_udbuildenv                     #Initialize the udbuild environment

                                    #Download the source file if it doesn't already exist
download $SITEURL/$PKGNAME-$VERSION.tar.gz
unpack $PKGNAME-$VERSION.tar.gz     #Unpack the source file into $PREFIX/src and cd there

create_valet_template               #Create a template file for valet in the attic (you
                                    #can't just copy this into place)

./configure --prefix=$PREFIX        #Run your normal configure, without
                                    #having to define your own PREFIX
                                    #variable, because 'init_udbuildenv' did
                                    #that for you.

make                                #normal make commands

udbuildmon make install             #wrap your 'make install' with the
                                    #'udbuildmon' function to log what files
                                    #and directories were changed.
</file>

It is imperitive to start udbuild scripts with the string ''#!/bin/bash -l''
because this instructs bash to setup the VALET system.

== medium ==
<file sh udbuild>
#!/bin/bash -l

PKGNAME=cdo
VERSION=1.9.4
SITEURL=https://code.mpimet.mpg.de/attachments/download/17374/

vpkg_devrequire udbuild/2 eccodes/2.8.0:threads proj/5.1.0
vpkg_devrequire netcdf/4.6.1 udunits/2.2.26 fftw/3.3.8
init_udbuildenv

create_valet_template

download $SITEURL/$PKGNAME-$VERSION.tar.gz
unpack $PKGNAME-$VERSION.tar.gz

aflag CONFIG --with-szlib="$SZIP_PREFIX"                
aflag CONFIG --with-hdf5="$HDF5_PREFIX"                 
aflag CONFIG --with-netcdf="$NETCDF_PREFIX"             
aflag CONFIG --with-eccodes="$ECCODES_PREFIX"   
aflag CONFIG --with-proj="$PROJ_PREFIX"                 
aflag CONFIG --with-udunits2="$UDUNITS_PREFIX"  
aflag CONFIG --with-threads=yes                         
aflag CONFIG --with-curl=yes                    
aflag CONFIG --with-libxml=yes

./configure --prefix="$PREFIX" $CONFIG

sed -i 's/#include <string>/#include <string.h>/' src/modules.cc

make

udbuildmon make install
</file>

In this example, we use ''vpkg_devrequire'' to specify additional dependencies
needed to build the ''cdo'' package.  ''PREFIX'', however, will still be set
to ''/opt/shared/cdo/1.6.4''.

== complex ==
<file sh udbuild>
#!/bin/bash -l

PKGNAME=hdf4
VERSION=4.2.13
SITEURL=https://support.hdfgroup.org/ftp/HDF/HDF_Current/src

PKGINFO='HDF4: Data Model, Library, & File Format'
URLINFO=http://www.hdfgroup.org/products/hdf4/
                                 
vpkg_devrequire udbuild szip/2.1.1
init_udbuildenv

create_valet_template
                                     
download $SITEURL/hdf-$VERSION.tar.bz2
unpack hdf-$VERSION.tar.bz2
                 
aflag CFLAGS -fPIC                           
if valet intel; then   
        aflag CFLAGS -qopt-jump-tables=large
fi             

aflag CONFIG --disable-netcdf
aflag CONFIG --with-szlib=$SZIP_PREFIX
                                     
# Make shared libraries (sans fortran support):
./configure --prefix="$PREFIX" --enable-shared --disable-fortran $CONFIG
        
make install

make clean

# Make fortran enabled HDF4:
./configure --prefix="$PREFIX" --disable-shared --enable-fortran $CONFIG

make install

</file>

In this more complicated example, we still need dependencies, but this time
one of them will affect the ''PREFIX'' variable.  The Intel64 compiler will
be used, and PREFIX will be set to ''/opt/shared/hdf4/4.2.10-intel64''.

Furthermore, specific ''CFLAGS'' changes will be made for this compiler.
This example also illustrates how the ''VERSION'' string can be used.  Here,
we would set additional flags for the ''./configure'' script if the ''VERSION''
string were set to ''4.2.10-sansnetcdf''.  These options allow one build file
to build multiple versions of a package, and with only minor changes near
the top of the script (namely to the ''VERSION'' variable and the
''init_udbuildenv'' command-line.

Another interesting thing we do here is to make sure the installation is
as complete as possible.  HDF4 does not support shared object files for
fortran libraries.  So, first we build the shared objects which are possible,
then we enable fortran and ensure the full compliment of archive ''.a'' files
are present.

== python ==
<file sh udbuild-cdf>
#!/bin/bash -l

PKGNAME=cdf
VERSION=20180613
PY_VER=3.6.5
NETCDF_VER=4.6.1

PKGINFO='Python NetCDF, HDF5, HDF4, and dependent Modules'
URLINFO=https://pypi.org/

vpkg_devrequire udbuild python/$PY_VER netcdf/$NETCDF_VER
init_udbuildenv python-addon

create_valet_template

ppath PATH        "$PREFIX/bin"
apath LD_RUN_PATH "$PREFIX/lib"

pip_install CDF cdflib netCDF4 h5netCDF wera2netcdf Puppy nco
pip_install python-hdf4 hdf5able hdf5pickle hdf5storage
pip_install dtt2hdf ascii2hdf5 h5json h5pyd hdf5_matlab_reader
pip_install HDFconvert mrr2c hdfdict LazyHDF5 simpletraj mdtraj
pip_install mriqc ncagg
pip_install 'MDAnalysis[analysis, AMBER]'
</file>

The python example above is used to install the cdf and related modules as an add-on for python version 3.6.5. It is also considered a complex example since it displays the use of the option ''python-addon'' for ''init_udbuildenv'' to initialize the udbuild environment and ensure a python VALET package is loaded, sets ''PREFIX'' appropriately for python version's add-on installation path, and utilizes the udbuild ''pip_install'' function to call pip with using the correct ''PREFIX''. To use the python add-on module bundle, you can setup a VALET package.  Below is an example VALET yaml package created from the ''create_valet_template'' function.  It should be trimmed, as szip, hdf4, and hdf5 are all dependencies of netcdf.  The dependencies section should look like the ''vpkg_devrequire'' calls of the udbuild script.

<file sh python-cdf.vpkg_yaml>
python-cdf:
  description:          Python NetCDF, HDF5, HDF4, and dependent Modules
  url:                  https://pypi.org/
  prefix:               /opt/shared/python/add-ons
  
  default-version:      "3.6.5:20180613"
  
  versions:
    "3.6.5:20180613":
      description:      cdf compiled with system compilers
      prefix:           python3.6.5/cdf/20180613
      dependencies:
        -               python/3.6.5
        -               szip/2.1.1
        -               hdf4/4.2.13
        -               hdf5/1.10.2
        -               netcdf/4.6.1
</file>