Differences

This shows you the differences between two versions of the page.

--- abstract:darwin:install_software:udbuild [2021-04-22 11:33] – created anita
+++ abstract:darwin:install_software:udbuild [2021-04-22 11:37] (current) – anita
@@ Line 16: / Line 16: @@
 environment.
+==== Filesystem ====
+Software is deployed to ''/opt/shared''.
+The ''udbuild'' system defaults to this ''/opt/shared''
+location.  However, this can be changed by setting the ''UDBUILD_HOME''
+environment variable before initializing the ubuild environment.  A good
+value for this environment variable for workgroup software is
+''$WORKDIR/sw'', and for personal software installation is
+''$HOME/sw''. Refer to [[:abstract:darwin:.install_software:workgroup-sw|workgroup software installs]] for help setting up your directories for workgroup storage.
+Beneath this directory should be an ''attic'' sub-directory for downloaded
+software bundles, optionally an ''add-ons'' directory for software with
+optional add-ons, and one sub-directory for each package installed.
+These sub-directories should always be in all lower-case letter.  One more
+layer down should be a directory for each version of the software installed.
+It is important understand that on a complex cluster like DARWIN, the same
+release of a software package may have multiple installations due to various
+compiler and dependency package requirements.  These directories are the
+software installation roots.
+Underneath the installation root should be a directory called ''src'', which
+is the un-packed source bundle.  Next to ''src'' should be any ''bin'',
+''lib'', ''share'', etc. directories neccessary for the final deployment.
+An illustrated example of the software directory structure is as such:
+  * opt
+    * shared
+      * atlas
+         * 3.10.3
+         * 3.10.3-intel
+         * attic
+           * ''udbuild''  <sup>// - build and install script for atlas//</sup>
+      * python
+         * 2.7.8
+         * 3.2.5
+         * add-ons
+           * python2.7.15
+             * mpi
+               * 20180613
+           * python3.2.5
+         * attic
+           * ''udbuild''   <sup>// - build and install script for python//</sup>
+==== Building ====
+When building software, the base directory structure (including the ''attic''
+directory) should be created by you before proceeding further.
+You should download the software source bundle into ''attic''.  Then,
+unpack the software bundle and rename the directory to ''src'' as
+above.  This provides consistency in finding the source bundle and the
+''udbuild'' file.
+Examples of builds are provided below (after the udbuild function
+documentation).
+=== udbuild functions ===
+== init_udbuildenv ==
+This function initializes the udbuild environment. It ensures that you have
+the required ''PKGNAME'' and ''VERSION'' environment variables defined, you do
+not have VALET packages loaded before ''udbuild'' in your VALET history (these
+might affect your build), sets compiler variables like CC, FC, etc., then
+finally sets your ''PREFIX'' and ''VERSION'' variables based on it's
+command-line.  These command-line options affect ''init_udbuildenv'':
+  * none - This is equivalent to not supplying any parameters
+  * python-addon - Ensure a python VALET package is loaded, and set PREFIX \
+    appropriately for that python version's add-on installation path
+  * r-addon - Ensure an R VALET package is loaded, and set PREFIX \
+    appropriately for that R version's add-on installation path
+  * node-addon - Ensure a Node.JS version is loaded and set the PREFIX \
+    appropriately for that Node.JS versions' add-on installation path
+  * Any other arguments are treated as the names of VALET packages which are \
+    loaded and added to the ''VERSION'' environment variable.
+After all of this, your ''PREFIX'' variable will be set to
+  ${UDBUILD_HOME:-/opt/shared}/$PKGNAME/$VERSION
+== debug ==
+Drop into a debug shell.  If the debug shell is exited cleanly, then the
+udbuild script will continue from there.  This is a useful routine to use
+when creating a udbuild script.  You may want to build the script based
+on documentation, and run a ''debug'' function between a configure and
+make step to verify the environment looks sane.  After the first successful
+compile, you can then remove the debug line.
+== download ==
+The ''download'' function takes a mandatory and optional argument.  The mandatory,
+first, argument is the URL to download.  The resulting file will be named after
+the last part of the URL unless the optional second argument is specified.  In this
+case, the resulting file will be named after the second argument.
+If a file with the same name already exists, the download exits successfully without
+doing anything.  If you wish to re-download the archive, delete or rename the existing
+one.
+== unpack ==
+The ''unpack'' function takes a mandatory and optional argument.  The mandatory,
+first, argument is the name of an archive file (tar.gz, tar.bz2, tar.xz, zip, etc.)
+to extract.  The archive will be unpacked into a directory named ''src'' under the
+install prefix (versioned directory for installation) unless the optional second argument
+is specified, then it will be used in place of the name ''src''.  This directory, and
+its parents if necessary, will be created prior to extraction.  Source archives using
+the ''tar'' format customarily have a single top-level directory entry which contains
+the package name and version, this is automatically stripped from the extracted archive.
+After completing the extraction process, the ''unpack'' function places the udbuild
+script into the newly created directory to prepare the script for configure and make
+steps.
+If the ''src'' (or alternately specified) directory exists, then the archive is not
+extracted over it.  In this case, the function returns successfully without doing
+anything.  If you wish to force a new extraction, remove or rename the existing
+''src'' directory.
+== create_valet_template ==
+Create a YAML based valet package file template and place it in the ''attic'' directory
+if one can be found, and the same directory as the udbuild script if one cannot.  This
+template is helpful, but usually cannot just be copied into place.  For example, it only
+knows about the version of the software it is installing, and copying the file blindly would
+remove entries for all other versions.  Furthermore, the "dependencies" entry is filled with
+all loaded valet packages, even if they are dependencies of dependencies, and not needed
+to be explicitly listed.
+== valet ==
+This function takes either the name of a package (e.g. ''openmpi''), or a
+package name/version pair (e.g. ''openmpi/1.8.2'') and return true if there
+is a VALET package loaded to satisfy this dependency, and false otherwise.
+This function can be used along with any other shell constructs, such as
+''if'' ... ''else'' ... ''fi'', to modify the behaviour of a build.
+== version ==
+This function takes a string and validates that it exists as a complete
+entry (i.e. starts, stops, or is bounded by hyphens) in the VERSION string.
+This function can be used along with any other shell constructs, such as
+''if'' ... ''else'' ... ''fi'', to modify the behavior of a build.
+== package ==
+This function takes a string and validates that it exists as part of
+the final package name, which may include features.  This is useful for
+matching features which are specified to configure a software build but
+don't show up in the version string or require a valet package be
+available, the string "threads" for OpenMP is a good example of using
+this feature.
+== ifvalet ==
+This function is shorthand for ''if valet "$1"; then shift; eval "$@"; fi''
+to make udbuild scripts simple to read and code.
+== ifversion ==
+This function is shorthand for ''if version "$1"; then shift; eval "$@"; fi''
+to make udbuild scripts simple to read and code.
+== ifpackage ==
+This function is shorthand for ''if package "$1"; then shift; eval "$@"; fi''
+to make udbuild scripts simple to read and code.
+== udbuildcapture ==
+Put all screen output into a capture file.  The main purpose of this is to
+log questions answered during an interactive isntall, to document what
+choices were made.
+== udbuildmon ==
+This script is helpful to be run during the install phase of a build, for
+example:
+  udbuildmon make install
+It will log all ''open'' for write and ''mkdir'' system calls and log them to
+a file named ''udbuildmon.log''.  You can use this log file to verify the
+build did not write any files to unknown locations.  This function should not
+be necessary with [[http://www.cmake.org|cmake]] builds, as they normally
+store this information in an ''install_manifest.txt'' file.
+== apath ==
+Append a path to a variable and ensure that variable is exported.  The required
+first argument is the environment variable name, all remaining arguments are
+paths to append to the end of the environment variable.  A colon (:) character
+is used as the delimiter, as is standard in path environment variables.
+== ppath ==
+Prepend a path to a variable similar to ''apath'', but instead of adding the
+path to the end, add it to the beginning.  Arguments are the same as ''apath''
+== rpath ==
+Remove a path from an environment variable.  The required first argument is the
+environment variable name.  All remaining arguments are removed from the
+environment variable.  If an entry exists multiple times, all instances are
+removed.
+== aflag ==
+Append a flag to an environment variable.  The required first argument is the
+environment variable name.  All remaining arguments are added to the environment
+variable.  The ''aflag'' variable works under two contexts, which depend on the
+status of the first argument.  If it is an already defined bash array variable
+type, then the remaining arguments are added as new elements in the array.  In
+all other cases, the remaining arguments are added to the string using a space
+character as a delimiter.
+Using a bash array has the advantage of allowing flags which contain whitespace
+characters.  If this is a requirement, the following steps should be undertaken:
+  CONFIG=()                            #Declare CONFIG as a bash array
+  aflag CONFIG --prefix="$PREFIX"      #Specify flags, this one is regular
+  aflag CONFIG --title="My Software"   #Specify flags, his one with spaces
+  ./configure  "${CONFIG[@]}"          #bash syntax for arrays as arguments
+== pflag ==
+Prepend flags to an environment variable.  This is the same as the ''aflag''
+function, but it puts its arguments at the beginning of the variable.  Its
+arguments are identical.
+== rflag ==
+Remove a flag from an environment variable.  This works similar to the
+''rpath'' variable and also supports bash arrays.
+== udexpect ==
+This is a wrapper around the TCL ''expect'' utility to simplify the process of
+answering questions for interactive builds.  This function accepts an expect
+script as STDIN (the normal method is via HERE-DOC) and provides all the basics
+of running ''expect''.  Some standard responses are provided to simplify the
+process:
+  - enter - Send a carrige return as if the user pressed their "Enter" key
+  - yes - Send the string "yes" as if the user typed "yes" and pressed "Enter"
+  - no - Send "no" and press "Enter"
+  - y - Send "y" and press "Enter"
+  - n - Send "n" and press "Enter"
+  - respond text - Send //text// and press "Enter"
+  - keypress c - Send the character //c// and DO NOT press "Enter"
+  - user - Prompt the person at the keyboard for a respone, and send it, press "Enter"
+== makeflags_set ==
+Update a file (presumably a Makefile and specified as the first argument) which uses
+the syntax "key=value" and update the value of the second argument to be that of the
+third argument.  This is a simple helper function to make it simple to edit basic
+information in a Makefile.
+== makeflags_prepend ==
+Update a file similar to ''makeflags_set'', except prepend the third argument to the
+existing value, instead of replacing it.
+== makeflags_append ==
+Update a file similar to ''makeflags_set'', except append the third argument to the
+existing value, instead of replacing it.
+=== udbuild script examples ===
+== simple ==
+In this example, an easy-to-install software package called cmake is built
+and isntalled.  It has no software dependencies, and uses the standard
+''configure'', ''make'', ''make install'' procedure used by very many
+open source software packages.
+To prepare for this build, you would want to create the following directories:
+<file sh udbuild>
+#/bin/bash -l
+PKGNAME=cmake                       #These are required variables and must
+VERSION=3.11.3                      #be set before calling 'init_udbuildenv'
+                                    #Setting the "SITEURL" is not required, but is
+                                    #helpful later.
+SITEURL=https://cmake.org/files/v${VERSION%.*}
+PKGINFO='Cross-Platform Make'       #Helpful for the ''create_valet_template'' function,
+URLINFO=http://www.cmake.org        #but not required at all
+vpkg_devrequire udbuild/2           #Use VALET to load the udbuild v2 environment
+init_udbuildenv                     #Initialize the udbuild environment
+                                    #Download the source file if it doesn't already exist
+download $SITEURL/$PKGNAME-$VERSION.tar.gz
+unpack $PKGNAME-$VERSION.tar.gz     #Unpack the source file into $PREFIX/src and cd there
+create_valet_template               #Create a template file for valet in the attic (you
+                                    #can't just copy this into place)
+./configure --prefix=$PREFIX        #Run your normal configure, without
+                                    #having to define your own PREFIX
+                                    #variable, because 'init_udbuildenv' did
+                                    #that for you.
+make                                #normal make commands
+udbuildmon make install             #wrap your 'make install' with the
+                                    #'udbuildmon' function to log what files
+                                    #and directories were changed.
+</file>
+It is imperitive to start udbuild scripts with the string ''#!/bin/bash -l''
+because this instructs bash to setup the VALET system.
+== medium ==
+<file sh udbuild>
+#!/bin/bash -l
+PKGNAME=cdo
+VERSION=1.9.4
+SITEURL=https://code.mpimet.mpg.de/attachments/download/17374/
+vpkg_devrequire udbuild/2 eccodes/2.8.0:threads proj/5.1.0
+vpkg_devrequire netcdf/4.6.1 udunits/2.2.26 fftw/3.3.8
+init_udbuildenv
+create_valet_template
+download $SITEURL/$PKGNAME-$VERSION.tar.gz
+unpack $PKGNAME-$VERSION.tar.gz
+aflag CONFIG --with-szlib="$SZIP_PREFIX"
+aflag CONFIG --with-hdf5="$HDF5_PREFIX"
+aflag CONFIG --with-netcdf="$NETCDF_PREFIX"
+aflag CONFIG --with-eccodes="$ECCODES_PREFIX"
+aflag CONFIG --with-proj="$PROJ_PREFIX"
+aflag CONFIG --with-udunits2="$UDUNITS_PREFIX"
+aflag CONFIG --with-threads=yes
+aflag CONFIG --with-curl=yes
+aflag CONFIG --with-libxml=yes
+./configure --prefix="$PREFIX" $CONFIG
+sed -i 's/#include <string>/#include <string.h>/' src/modules.cc
+make
+udbuildmon make install
+</file>
+In this example, we use ''vpkg_devrequire'' to specify additional dependencies
+needed to build the ''cdo'' package.  ''PREFIX'', however, will still be set
+to ''/opt/shared/cdo/1.6.4''.
+== complex ==
+<file sh udbuild>
+#!/bin/bash -l
+PKGNAME=hdf4
+VERSION=4.2.13
+SITEURL=https://support.hdfgroup.org/ftp/HDF/HDF_Current/src
+PKGINFO='HDF4: Data Model, Library, & File Format'
+URLINFO=http://www.hdfgroup.org/products/hdf4/
+vpkg_devrequire udbuild szip/2.1.1
+init_udbuildenv
+create_valet_template
+download $SITEURL/hdf-$VERSION.tar.bz2
+unpack hdf-$VERSION.tar.bz2
+aflag CFLAGS -fPIC
+if valet intel; then
+        aflag CFLAGS -qopt-jump-tables=large
+fi
+aflag CONFIG --disable-netcdf
+aflag CONFIG --with-szlib=$SZIP_PREFIX
+# Make shared libraries (sans fortran support):
+./configure --prefix="$PREFIX" --enable-shared --disable-fortran $CONFIG
+make install
+make clean
+# Make fortran enabled HDF4:
+./configure --prefix="$PREFIX" --disable-shared --enable-fortran $CONFIG
+make install
+</file>
+In this more complicated example, we still need dependencies, but this time
+one of them will affect the ''PREFIX'' variable.  The Intel64 compiler will
+be used, and PREFIX will be set to ''/opt/shared/hdf4/4.2.10-intel64''.
+Furthermore, specific ''CFLAGS'' changes will be made for this compiler.
+This example also illustrates how the ''VERSION'' string can be used.  Here,
+we would set additional flags for the ''./configure'' script if the ''VERSION''
+string were set to ''4.2.10-sansnetcdf''.  These options allow one build file
+to build multiple versions of a package, and with only minor changes near
+the top of the script (namely to the ''VERSION'' variable and the
+''init_udbuildenv'' command-line.
+Another interesting thing we do here is to make sure the installation is
+as complete as possible.  HDF4 does not support shared object files for
+fortran libraries.  So, first we build the shared objects which are possible,
+then we enable fortran and ensure the full compliment of archive ''.a'' files
+are present.
+== python ==
+<file sh udbuild-cdf>
+#!/bin/bash -l
+PKGNAME=cdf
+VERSION=20180613
+PY_VER=3.6.5
+NETCDF_VER=4.6.1
+PKGINFO='Python NetCDF, HDF5, HDF4, and dependent Modules'
+URLINFO=https://pypi.org/
+vpkg_devrequire udbuild python/$PY_VER netcdf/$NETCDF_VER
+init_udbuildenv python-addon
+create_valet_template
+ppath PATH        "$PREFIX/bin"
+apath LD_RUN_PATH "$PREFIX/lib"
+pip_install CDF cdflib netCDF4 h5netCDF wera2netcdf Puppy nco
+pip_install python-hdf4 hdf5able hdf5pickle hdf5storage
+pip_install dtt2hdf ascii2hdf5 h5json h5pyd hdf5_matlab_reader
+pip_install HDFconvert mrr2c hdfdict LazyHDF5 simpletraj mdtraj
+pip_install mriqc ncagg
+pip_install 'MDAnalysis[analysis, AMBER]'
+</file>
+The python example above is used to install the cdf and related modules as an add-on for python version 3.6.5. It is also considered a complex example since it displays the use of the option ''python-addon'' for ''init_udbuildenv'' to initialize the udbuild environment and ensure a python VALET package is loaded, sets ''PREFIX'' appropriately for python version's add-on installation path, and utilizes the udbuild ''pip_install'' function to call pip with using the correct ''PREFIX''. To use the python add-on module bundle, you can setup a VALET package.  Below is an example VALET yaml package created from the ''create_valet_template'' function.  It should be trimmed, as szip, hdf4, and hdf5 are all dependencies of netcdf.  The dependencies section should look like the ''vpkg_devrequire'' calls of the udbuild script.
+<file sh python-cdf.vpkg_yaml>
+python-cdf:
+  description:          Python NetCDF, HDF5, HDF4, and dependent Modules
+  url:                  https://pypi.org/
+  prefix:               /opt/shared/python/add-ons
+  default-version:      "3.6.5:20180613"
+  versions:
+    "3.6.5:20180613":
+      description:      cdf compiled with system compilers
+      prefix:           python3.6.5/cdf/20180613
+      dependencies:
+        -               python/3.6.5
+        -               szip/2.1.1
+        -               hdf4/4.2.13
+        -               hdf5/1.10.2
+        -               netcdf/4.6.1
+</file>