Differences

This shows you the differences between two versions of the page.

--- abstract:caviness:install_software [2018-07-09 16:43] – external edit 127.0.0.1
+++ abstract:caviness:install_software [2018-08-23 10:26] (current) – removed anita
@@ Line 1: / Line 1: @@
-===== UDBUILD Software Deployment =====
-The software installed and deployed on the Caviness cluster each has it's own
-methods for compiling and installing.  To manage this process, the HPC team
-has a set of standards and technology used to reduce complexity and bring
-consistency to the process.
-Software is built, installed, and accessed using the
-[[software:valet:valet|VALET]] system developed by Dr. Jeffrey Frey.  IT
-developed a set of software helper functions which can be access using VALET
-by importing the ''udbuild'' vpkg.
-This page describes the filesystem layout used by IT, and the anatomy of the
-''udbuild'' file used to build and deploy software.  Throughout this process,
-it is helpful to have an understanding of how to use
-[[software:valet:valet|VALET]] to add and remove software packages from your
-environment.
-==== Filesystem ====
-Software is deployed to ''/opt/shared''.
-The ''udbuild'' system defaults to this ''/opt/shared''
-location.  However, this can be changed by setting the ''UDBUILD_HOME''
-environment variable before initializing the ubuild environment.  A good
-value for this environment variable for workgroup software is
-''$WORKDIR/sw'', and for personal software installation is
-''$HOME/sw''.
-Beneath this directory should be an ''attic'' sub-directory for downloaded
-software bundles, optionally an ''add-ons'' directory for software with
-optional add-ons, and one sub-directory for each package installed.
-These sub-directories should always be in all lower-case letter.  One more
-layer down should be a directory for each version of the software installed.
-It is important understand that on a complex cluster like Caviness, the same
-release of a software package may have multiple installations due to various
-compiler and dependency package requirements.  These directories are the
-software installation roots.
-Underneath the installation root should be a directory called ''src'', which
-is the un-packed source bundle.  Next to ''src'' should be any ''bin'',
-''lib'', ''share'', etc. directories neccessary for the final deployment.
-An illustrated example of the software directory structure is as such:
-  * opt
-    * shared
-      * atlas
-         * 3.10.2
-           * lib
-           * include
-           * src
-             * ''udbuild''   <sup>// - build and install script for atlas 3.10.2//</sup>
-         * 3.10.2-intel64
-         * attic
-      * python
-         * 2.7.8
-           * bin
-           * include
-           * lib
-           * share
-           * src
-             * ''udbuild''   <sup>// - build and install script for python 2.7.8//</sup>
-         * 3.2.5
-         * add-ons
-           * python2.7.8
-             * numpy
-               * 1.8.2
-                 * bin
-                 * lib
-                 * src
-                   * ''udbuild''  <sup>// - build and install script for the numpy 1.8.2 installed for python 2.7.8//</sup>
-             * ipython
-           * python3.2.5
-         * attic
-==== Building ====
-When building software, the base directory structure (including the ''attic''
-directory) should be created by you before proceeding further.
-You should download the software source bundle into ''attic''.  Then,
-unpack the software bundle and rename the directory to ''src'' as
-above.  This provides consistency in finding the source bundle and the
-''udbuild'' file.
-Examples of builds are provided below (after the udbuild function
-documentation).
-=== udbuild functions ===
-== init_udbuildenv ==
-This function initializes the udbuild environment. It ensures that you have
-the required ''PKGNAME'' and ''VERSION'' environment variables defined, you do
-not have VALET packages loaded before ''udbuild'' in your VALET history (these
-might affect your build), sets compiler variables like CC, FC, etc., then
-finally sets your ''PREFIX'' and ''VERSION'' variables based on it's
-command-line.  These command-line options affect ''init_udbuildenv'':
-  * none - This is equivalent to not supplying any parameters
-  * python-addon - Ensure a python VALET package is loaded, and set PREFIX \
-    appropriately for that python version's add-on installation path
-  * r-addon - Ensure an R VALET package is loaded, and set PREFIX \
-    appropriately for that R version's add-on installation path
-  * Any other arguments are treated as the names of VALET packages which are \
-    loaded and added to the ''VERSION'' environment variable.
-After all of this, your ''PREFIX'' variable will be set to
-  ${UDBUILD_HOME:-/opt/shared}/$PKGNAME/$VERSION
-== valet ==
-This function takes either the name of a package (e.g. ''openmpi''), or a
-package name/version pair (e.g. ''openmpi/1.8.2'') and return true if there
-is a VALET package loaded to satisfy this dependency, and false otherwise.
-This function can be used along with any other shell constructs, such as
-''if'' ... ''else'' ... ''fi'', to modify the behaviour of a build.
-== version ==
-This function takes a string and validates that it exists as a complete
-entry (i.e. starts, stops, or is bounded by hyphens) in the VERSION string.
-This function can be used along with any other shell constructs, such as
-''if'' ... ''else'' ... ''fi'', to modify the behaviour of a build.
-== ifvalet ==
-This function is shorthand for ''if valet "$1"; then shift; eval "$@"; fi''
-to make udbuild scripts simple to read and code.
-== ifversion ==
-This function is shorthand for ''if version "$1"; then shift; eval "$@"; fi''
-to make udbuild scripts simple to read and code.
-== udbuildcapture ==
-Put all screen output into a capture file.  The main purpose of this is to
-log questions answered during an interactive isntall, to document what
-choices were made.
-== udbuildmon ==
-This script is helpful to be run during the install phase of a build, for
-example:
-  udbuildmon make install
-It will log all ''open'' for write and ''mkdir'' system calls and log them to
-a file named ''udbuildmon.log''.  You can use this log file to verify the
-build did not write any files to unknown locations.  This function should not
-be necessary with [[http://www.cmake.org|cmake]] builds, as they normally
-store this information in an ''install_manifest.txt'' file.
-=== udbuild script examples ===
-== simple ==
-In this example, an easy-to-install software package called cmake is built
-and isntalled.  It has no software dependencies, and uses the standard
-''configure'', ''make'', ''make install'' procedure used by very many
-open source software packages.
-To prepare for this build, you would want to create the following directories:
-<file sh udbuild>
-#/bin/bash -l
-PKGNAME=cmake                       #These are required variables and must
-VERSION=3.0.1                       #be set before calling 'init_udbuildenv'
-vpkg_devrequire udbuild             #Use VALET to load the udbuild environment
-init_udbuildenv                     #Initialize the udbuild environment
-./configure --prefix=$PREFIX        #Run your normal configure, without
-                                    #having to define your own PREFIX
-                                    #variable, because 'init_udbuildenv' did
-                                    #that for you.
-make                                #normal make commands
-udbuildmon make install             #wrap your 'make install' with the
-                                    #'udbuildmon' function to log what files
-                                    #and directories were changed.
-</file>
-It is imperitive to start udbuild scripts with the string ''#!/bin/bash -l''
-because this instructs bash to setup the VALET system.
-== medium ==
-<file sh udbuild>
-#!/bin/bash -l
-PKGNAME=cdo
-VERSION=1.6.4
-vpkg_devrequire udbuild
-vpkg_devrequire netcdf/4.3.2 grib/1.12.3 udunits/2.2.17 proj/4 fftw/3
-init_udbuildenv
-./configure --prefix="$PREFIX"                          \
-                --with-szlib="$SZIP_PREFIX"             \
-                --with-hdf5="$HDF5_PREFIX"              \
-                --with-netcdf="$NETCDF_PREFIX"          \
-                --with-grib_api="$GRIB_PREFIX"          \
-                --with-proj="$PROJ_PREFIX"              \
-                --with-udunits2="$UDUNITS_PREFIX"       \
-                --with-threads=yes                      \
-                --with-curl=yes                         \
-                --with-libxml=yes
-make
-udbuildmon make install
-</file>
-In this example, we use ''vpkg_devrequire'' to specify additional dependencies
-needed to build the ''cdo'' package.  ''PREFIX'', however, will still be set
-to ''/opt/shared/cdo/1.6.4''.
-== complex ==
-<file sh udbuild>
-#!/bin/bash -l
-PKGNAME=hdf4
-VERSION=4.2.10
-vpkg_devrequire udbuild szip/2.1
-init_udbuildenv intel64
-CFLAGS="$CFLAGS -fPIC"
-CONFIGURE_FLAGS=''
-ifvalet intel64         'CFLAGS="$CFLAGS -qopt-jump-tables=large"'
-ifversion sansnetcdf    'CONFIGURE_FLAGS="--disable-netcdf"'
-# Make shared libraries (sans fortran support):
-./configure --prefix="$PREFIX" --enable-shared --disable-fortran \
-        --with-szlib=$SZIP_PREFIX $CONFIGURE_FLAGS
-udbuildmon make install
-make clean
-# Make fortran enabled HDF4:
-./configure --prefix="$PREFIX" --disable-shared --enable-fortran \
-        --with-szlib=$SZIP_PREFIX $CONFIGURE_FLAGS
-udbuildmon make install
-</file>
-In this more complicated example, we still need dependencies, but this time
-one of them will affect the ''PREFIX'' variable.  The Intel64 compiler will
-be used, and PREFIX will be set to ''/opt/shared/hdf4/4.2.10-intel64''.
-Furthermore, specific ''CFLAGS'' changes will be made for this compiler.
-This example also illustrates how the ''VERSION'' string can be used.  Here,
-we would set additional flags for the ''./configure'' script if the ''VERSION''
-string were set to ''4.2.10-sansnetcdf''.  These options allow one build file
-to build multiple versions of a package, and with only minor changes near
-the top of the script (namely to the ''VERSION'' variable and the
-''init_udbuildenv'' command-line.
-Another interesting thing we do here is to make sure the installation is
-as complete as possible.  HDF4 does not support shared object files for
-fortran libraries.  So, first we build the shared objects which are possible,
-then we enable fortran and ensure the full compliment of archive ''.a'' files
-are present.
-== python ==
-<file sh udbuild>
-#!/bin/bash -l
-UDBUILD_HOME=$WORKDIR/sw
-PKGNAME=netcdf
-# Change this to the version of netcdf
-VERSION=1.1.7.1
-# Change this to the version of python
-PY_VERS=2.7.8
-# Load the udbuild package
-vpkg_devrequire udbuild
-# These are all the dependencies for netcdf specifying the
-# particular python version based on PY_VERS environment variable
-vpkg_devrequire python/$PY_VERS
-vpkg_devrequire python-numpy/python$PY_VERS
-vpkg_devrequire netcdf/4.3
-# This initializes the environment for the build based on the
-vpkg_devrequire python-numpy/python$PY_VERS
-vpkg_devrequire netcdf/4.3
-# This initializes the environment for the build based on the
-# package name and version, and we use the install option
-# for a python add-on (python-addon)
-init_udbuildenv python-addon
-# These are environment variables needed for the install - hints for
-# what might be needed can be found in the setup.cfg.template file
-export HDF5_DIR=${HDF5_PREFIX}
-export NETCDF4_DIR=${NETCDF_PREFIX}
-export HDF4_DIR=${HDF4_PREFIX}
-# Standard install is 'python setup.py build' but we use $PYTHON to get
-# the correct version of python we are building the add-on for as defined
-# above rather than a default system version
-$PYTHON setup.py build
-$PYTHON setup.py install --prefix=$PREFIX
-</file>
-The python example above is used to install the netcdf library version 1.1.7.1 add-on for python version 2.7.8 in the shared research software directory ''$WORKDIR/sw'' following the directory setup described for [[#filesystem]] above. It is also considered a complex example since it displays the use of the option ''python-addon'' for ''init_udbuildenv'' to initialize the udbuild environment and ensure a python VALET package is loaded, sets ''PREFIX'' appropriately for python version's add-on installation path, and utilizes ''$PYTHON'' to get the correct version of python for the install. You can build and install the same version of netcdf 1.1.7.1 for a different version of python by simply changing ''PY_VERS'' to the correct version and rerunning your udbuild script. To use the python add-on, you can setup a VALET package.  Below is an example VALET 2.0 json package called ''python-netcdf'' based on netcdf version 1.1.7.1 installed for both python versions 2.7.8 and 3.2.5. You need to change ''<WORKGROUP_NAME>'' to match the corresponding shared research software directory workgroup name where you installed the python add-on software.
-<file sh python-netcdf.vpkg_json>
-#
-# This is the python-netcdf package for VALET 2.0
-# supporting json package style.
-#
-{
-  "python-netcdf": {
-# Change to your install location (i.e. <WORKGROUP_NAME>)
-    "prefix":           "/home/work/<WORKGROUP_NAME>/sw/python/add-ons",
-    "url":              "https://pypi.python.org/pypi/netCDF4",
-    "description":      "Python NetCDF",
-    "default-version":  "1.7.1.1-python2.7.8",
-    "standard-paths":   true,
-    "development-env":  false,
-    "actions": [
-      {
-        "action":       "path-prepend",
-        "variable":     "PYTHONPATH",
-        "value":        "${VALET_PATH_PREFIX}/lib/python${PYTHON_VERSION_SHORT}/site-packages"
-      }
-    ],
-    "versions": {
-      "python2.7.8": {
-        "alias-to":     "1.1.7.1-python2.7.8"
-      },
-      "python3.2.5": {
-        "alias-to":     "1.1.7.1-python3.2.5"
-      },
-      "1.1.7.1-python2.7.8": {
-        "description":  "Version 1.1.7.1 and python 2.7.8",
-        "prefix":       "python2.7.8/netcdf/1.1.7.1",
-        "dependencies": [
-                        "python/2.7.8",
-                        "python-numpy/python2.7.8",
-        "prefix":       "python2.7.8/netcdf/1.1.7.1",
-        "dependencies": [
-                        "python/2.7.8",
-                        "python-numpy/python2.7.8",
-                        "netcdf/4.3"
-        ]
-      },
-      "1.1.7.1-python3.2.5": {
-        "description":  "Version 1.1.7.1 and python 3.2.5",
-        "prefix":       "python3.2.5/netcdf/1.1.7.1",
-        "dependencies": [
-                        "python/3.2.5",
-                        "python-numpy/python3.2.5",
-                        "netcdf/4.3"
-        ]
-      }
-    }
-  }
-}
-</file>