Show pageOld revisionsBack to top This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong. ===== UDBUILD Software Deployment ===== The software installed and deployed on the Farber cluster each has it's own methods for compiling and installing. To manage this process, the HPC team has a set of standards and technology used to reduce complexity and bring consistency to the process. Software is built, installed, and accessed using the [[software:valet:valet|VALET]] system developed by Dr. Jeffrey Frey. IT developed a set of software helper functions which can be access using VALET by importing the ''udbuild'' vpkg. This page describes the filesystem layout used by IT, and the anatomy of the ''udbuild'' file used to build and deploy software. Throughout this process, it is helpful to have an understanding of how to use [[software:valet:valet|VALET]] to add and remove software packages from your environment. ==== Filesystem ==== Software is deployed to ''/opt/shared''. The ''udbuild'' system defaults to this ''/opt/shared'' location. However, this can be changed by setting the ''UDBUILD_HOME'' environment variable before initializing the ubuild environment. A good value for this environment variable for workgroup software is ''$WORKDIR/sw'', and for personal software installation is ''$HOME/sw''. Refer to [[:technical:developer:workgroup-sw|workgroup software installs]] for help setting up your directories for workgroup storage. Beneath this directory should be an ''attic'' sub-directory for downloaded software bundles, optionally an ''add-ons'' directory for software with optional add-ons, and one sub-directory for each package installed. These sub-directories should always be in all lower-case letter. One more layer down should be a directory for each version of the software installed. It is important understand that on a complex cluster like Farber, the same release of a software package may have multiple installations due to various compiler and dependency package requirements. These directories are the software installation roots. Underneath the installation root should be a directory called ''src'', which is the un-packed source bundle. Next to ''src'' should be any ''bin'', ''lib'', ''share'', etc. directories neccessary for the final deployment. An illustrated example of the software directory structure is as such: * opt * shared * atlas * 3.10.2 * lib * include * src * ''udbuild'' <sup>// - build and install script for atlas 3.10.2//</sup> * 3.10.2-intel64 * attic * python * 2.7.8 * bin * include * lib * share * src * ''udbuild'' <sup>// - build and install script for python 2.7.8//</sup> * 3.2.5 * add-ons * python2.7.8 * numpy * 1.8.2 * bin * lib * src * ''udbuild'' <sup>// - build and install script for the numpy 1.8.2 installed for python 2.7.8//</sup> * ipython * python3.2.5 * attic ==== Building ==== When building software, the base directory structure (including the ''attic'' directory) should be created by you before proceeding further. You should download the software source bundle into ''attic''. Then, unpack the software bundle and rename the directory to ''src'' as above. This provides consistency in finding the source bundle and the ''udbuild'' file. Examples of builds are provided below (after the udbuild function documentation). === udbuild functions === == init_udbuildenv == This function initializes the udbuild environment. It ensures that you have the required ''PKGNAME'' and ''VERSION'' environment variables defined, you do not have VALET packages loaded before ''udbuild'' in your VALET history (these might affect your build), sets compiler variables like CC, FC, etc., then finally sets your ''PREFIX'' and ''VERSION'' variables based on it's command-line. These command-line options affect ''init_udbuildenv'': * none - This is equivalent to not supplying any parameters * python-addon - Ensure a python VALET package is loaded, and set PREFIX \ appropriately for that python version's add-on installation path * r-addon - Ensure an R VALET package is loaded, and set PREFIX \ appropriately for that R version's add-on installation path * Any other arguments are treated as the names of VALET packages which are \ loaded and added to the ''VERSION'' environment variable. After all of this, your ''PREFIX'' variable will be set to ${UDBUILD_HOME:-/opt/shared}/$PKGNAME/$VERSION == valet == This function takes either the name of a package (e.g. ''openmpi''), or a package name/version pair (e.g. ''openmpi/1.8.2'') and return true if there is a VALET package loaded to satisfy this dependency, and false otherwise. This function can be used along with any other shell constructs, such as ''if'' ... ''else'' ... ''fi'', to modify the behaviour of a build. == version == This function takes a string and validates that it exists as a complete entry (i.e. starts, stops, or is bounded by hyphens) in the VERSION string. This function can be used along with any other shell constructs, such as ''if'' ... ''else'' ... ''fi'', to modify the behaviour of a build. == ifvalet == This function is shorthand for ''if valet "$1"; then shift; eval "$@"; fi'' to make udbuild scripts simple to read and code. == ifversion == This function is shorthand for ''if version "$1"; then shift; eval "$@"; fi'' to make udbuild scripts simple to read and code. == udbuildcapture == Put all screen output into a capture file. The main purpose of this is to log questions answered during an interactive isntall, to document what choices were made. == udbuildmon == This script is helpful to be run during the install phase of a build, for example: udbuildmon make install It will log all ''open'' for write and ''mkdir'' system calls and log them to a file named ''udbuildmon.log''. You can use this log file to verify the build did not write any files to unknown locations. This function should not be necessary with [[http://www.cmake.org|cmake]] builds, as they normally store this information in an ''install_manifest.txt'' file. === udbuild script examples === == simple == In this example, an easy-to-install software package called cmake is built and isntalled. It has no software dependencies, and uses the standard ''configure'', ''make'', ''make install'' procedure used by very many open source software packages. To prepare for this build, you would want to create the following directories: <file sh udbuild> #/bin/bash -l PKGNAME=cmake #These are required variables and must VERSION=3.0.1 #be set before calling 'init_udbuildenv' vpkg_devrequire udbuild #Use VALET to load the udbuild environment init_udbuildenv #Initialize the udbuild environment ./configure --prefix=$PREFIX #Run your normal configure, without #having to define your own PREFIX #variable, because 'init_udbuildenv' did #that for you. make #normal make commands udbuildmon make install #wrap your 'make install' with the #'udbuildmon' function to log what files #and directories were changed. </file> It is imperitive to start udbuild scripts with the string ''#!/bin/bash -l'' because this instructs bash to setup the VALET system. == medium == <file sh udbuild> #!/bin/bash -l PKGNAME=cdo VERSION=1.6.4 vpkg_devrequire udbuild vpkg_devrequire netcdf/4.3.2 grib/1.12.3 udunits/2.2.17 proj/4 fftw/3 init_udbuildenv ./configure --prefix="$PREFIX" \ --with-szlib="$SZIP_PREFIX" \ --with-hdf5="$HDF5_PREFIX" \ --with-netcdf="$NETCDF_PREFIX" \ --with-grib_api="$GRIB_PREFIX" \ --with-proj="$PROJ_PREFIX" \ --with-udunits2="$UDUNITS_PREFIX" \ --with-threads=yes \ --with-curl=yes \ --with-libxml=yes make udbuildmon make install </file> In this example, we use ''vpkg_devrequire'' to specify additional dependencies needed to build the ''cdo'' package. ''PREFIX'', however, will still be set to ''/opt/shared/cdo/1.6.4''. == complex == <file sh udbuild> #!/bin/bash -l PKGNAME=hdf4 VERSION=4.2.10 vpkg_devrequire udbuild szip/2.1 init_udbuildenv intel64 CFLAGS="$CFLAGS -fPIC" CONFIGURE_FLAGS='' ifvalet intel64 'CFLAGS="$CFLAGS -qopt-jump-tables=large"' ifversion sansnetcdf 'CONFIGURE_FLAGS="--disable-netcdf"' # Make shared libraries (sans fortran support): ./configure --prefix="$PREFIX" --enable-shared --disable-fortran \ --with-szlib=$SZIP_PREFIX $CONFIGURE_FLAGS udbuildmon make install make clean # Make fortran enabled HDF4: ./configure --prefix="$PREFIX" --disable-shared --enable-fortran \ --with-szlib=$SZIP_PREFIX $CONFIGURE_FLAGS udbuildmon make install </file> In this more complicated example, we still need dependencies, but this time one of them will affect the ''PREFIX'' variable. The Intel64 compiler will be used, and PREFIX will be set to ''/opt/shared/hdf4/4.2.10-intel64''. Furthermore, specific ''CFLAGS'' changes will be made for this compiler. This example also illustrates how the ''VERSION'' string can be used. Here, we would set additional flags for the ''./configure'' script if the ''VERSION'' string were set to ''4.2.10-sansnetcdf''. These options allow one build file to build multiple versions of a package, and with only minor changes near the top of the script (namely to the ''VERSION'' variable and the ''init_udbuildenv'' command-line. Another interesting thing we do here is to make sure the installation is as complete as possible. HDF4 does not support shared object files for fortran libraries. So, first we build the shared objects which are possible, then we enable fortran and ensure the full compliment of archive ''.a'' files are present. == python == <file sh udbuild> #!/bin/bash -l UDBUILD_HOME=$WORKDIR/sw PKGNAME=netcdf # Change this to the version of netcdf VERSION=1.1.7.1 # Change this to the version of python PY_VERS=2.7.8 # Load the udbuild package vpkg_devrequire udbuild # These are all the dependencies for netcdf specifying the # particular python version based on PY_VERS environment variable vpkg_devrequire python/$PY_VERS vpkg_devrequire python-numpy/python$PY_VERS vpkg_devrequire netcdf/4.3 # This initializes the environment for the build based on the # package name and version, and we use the install option # for a python add-on (python-addon) init_udbuildenv python-addon # These are environment variables needed for the install - hints for # what might be needed can be found in the setup.cfg.template file export HDF5_DIR=${HDF5_PREFIX} export NETCDF4_DIR=${NETCDF_PREFIX} export HDF4_DIR=${HDF4_PREFIX} # Standard install is 'python setup.py build' but we use $PYTHON to get # the correct version of python we are building the add-on for as defined # above rather than a default system version $PYTHON setup.py build $PYTHON setup.py install --prefix=$PREFIX </file> The python example above is used to install the netcdf library version 1.1.7.1 add-on for python version 2.7.8 in the shared research software directory ''$WORKDIR/sw'' following the directory setup described for [[#filesystem]] above. It is also considered a complex example since it displays the use of the option ''python-addon'' for ''init_udbuildenv'' to initialize the udbuild environment and ensure a python VALET package is loaded, sets ''PREFIX'' appropriately for python version's add-on installation path, and utilizes ''$PYTHON'' to get the correct version of python for the install. You can build and install the same version of netcdf 1.1.7.1 for a different version of python by simply changing ''PY_VERS'' to the correct version and rerunning your udbuild script. To use the python add-on, you can setup a VALET package. Below is an example VALET 2.0 json package called ''python-netcdf'' based on netcdf version 1.1.7.1 installed for both python versions 2.7.8 and 3.2.5. You need to change ''<WORKGROUP_NAME>'' to match the corresponding shared research software directory workgroup name where you installed the python add-on software. <file sh python-netcdf.vpkg_json> # # This is the python-netcdf package for VALET 2.0 # supporting json package style. # { "python-netcdf": { # Change to your install location (i.e. <WORKGROUP_NAME>) "prefix": "/home/work/<WORKGROUP_NAME>/sw/python/add-ons", "url": "https://pypi.python.org/pypi/netCDF4", "description": "Python NetCDF", "default-version": "1.7.1.1-python2.7.8", "standard-paths": true, "development-env": false, "actions": [ { "action": "path-prepend", "variable": "PYTHONPATH", "value": "${VALET_PATH_PREFIX}/lib/python${PYTHON_VERSION_SHORT}/site-packages" } ], "versions": { "python2.7.8": { "alias-to": "1.1.7.1-python2.7.8" }, "python3.2.5": { "alias-to": "1.1.7.1-python3.2.5" }, "1.1.7.1-python2.7.8": { "description": "Version 1.1.7.1 and python 2.7.8", "prefix": "python2.7.8/netcdf/1.1.7.1", "dependencies": [ "python/2.7.8", "python-numpy/python2.7.8", "prefix": "python2.7.8/netcdf/1.1.7.1", "dependencies": [ "python/2.7.8", "python-numpy/python2.7.8", "netcdf/4.3" ] }, "1.1.7.1-python3.2.5": { "description": "Version 1.1.7.1 and python 3.2.5", "prefix": "python3.2.5/netcdf/1.1.7.1", "dependencies": [ "python/3.2.5", "python-numpy/python3.2.5", "netcdf/4.3" ] } } } } </file> abstract/farber/install_software.txt Last modified: 2018-09-10 13:27by anita