Show pageOld revisionsBack to top This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong. ===== UDBUILD Software Deployment ===== The software installed and deployed on the DARWIN cluster each has it's own methods for compiling and installing. To manage this process, the HPC team has a set of standards and technology used to reduce complexity and bring consistency to the process. Software is built, installed, and accessed using the [[software:valet:valet|VALET]] system developed by Dr. Jeffrey Frey. IT developed a set of software helper functions which can be access using VALET by importing the ''udbuild'' vpkg. This page describes the filesystem layout used by IT, and the anatomy of the ''udbuild'' file used to build and deploy software. Throughout this process, it is helpful to have an understanding of how to use [[software:valet:valet|VALET]] to add and remove software packages from your environment. ==== Filesystem ==== Software is deployed to ''/opt/shared''. The ''udbuild'' system defaults to this ''/opt/shared'' location. However, this can be changed by setting the ''UDBUILD_HOME'' environment variable before initializing the ubuild environment. A good value for this environment variable for workgroup software is ''$WORKDIR/sw'', and for personal software installation is ''$HOME/sw''. Refer to [[:abstract:darwin:.install_software:workgroup-sw|workgroup software installs]] for help setting up your directories for workgroup storage. Beneath this directory should be an ''attic'' sub-directory for downloaded software bundles, optionally an ''add-ons'' directory for software with optional add-ons, and one sub-directory for each package installed. These sub-directories should always be in all lower-case letter. One more layer down should be a directory for each version of the software installed. It is important understand that on a complex cluster like DARWIN, the same release of a software package may have multiple installations due to various compiler and dependency package requirements. These directories are the software installation roots. Underneath the installation root should be a directory called ''src'', which is the un-packed source bundle. Next to ''src'' should be any ''bin'', ''lib'', ''share'', etc. directories neccessary for the final deployment. An illustrated example of the software directory structure is as such: * opt * shared * atlas * 3.10.3 * 3.10.3-intel * attic * ''udbuild'' <sup>// - build and install script for atlas//</sup> * python * 2.7.8 * 3.2.5 * add-ons * python2.7.15 * mpi * 20180613 * python3.2.5 * attic * ''udbuild'' <sup>// - build and install script for python//</sup> ==== Building ==== When building software, the base directory structure (including the ''attic'' directory) should be created by you before proceeding further. You should download the software source bundle into ''attic''. Then, unpack the software bundle and rename the directory to ''src'' as above. This provides consistency in finding the source bundle and the ''udbuild'' file. Examples of builds are provided below (after the udbuild function documentation). === udbuild functions === == init_udbuildenv == This function initializes the udbuild environment. It ensures that you have the required ''PKGNAME'' and ''VERSION'' environment variables defined, you do not have VALET packages loaded before ''udbuild'' in your VALET history (these might affect your build), sets compiler variables like CC, FC, etc., then finally sets your ''PREFIX'' and ''VERSION'' variables based on it's command-line. These command-line options affect ''init_udbuildenv'': * none - This is equivalent to not supplying any parameters * python-addon - Ensure a python VALET package is loaded, and set PREFIX \ appropriately for that python version's add-on installation path * r-addon - Ensure an R VALET package is loaded, and set PREFIX \ appropriately for that R version's add-on installation path * node-addon - Ensure a Node.JS version is loaded and set the PREFIX \ appropriately for that Node.JS versions' add-on installation path * Any other arguments are treated as the names of VALET packages which are \ loaded and added to the ''VERSION'' environment variable. After all of this, your ''PREFIX'' variable will be set to ${UDBUILD_HOME:-/opt/shared}/$PKGNAME/$VERSION == debug == Drop into a debug shell. If the debug shell is exited cleanly, then the udbuild script will continue from there. This is a useful routine to use when creating a udbuild script. You may want to build the script based on documentation, and run a ''debug'' function between a configure and make step to verify the environment looks sane. After the first successful compile, you can then remove the debug line. == download == The ''download'' function takes a mandatory and optional argument. The mandatory, first, argument is the URL to download. The resulting file will be named after the last part of the URL unless the optional second argument is specified. In this case, the resulting file will be named after the second argument. If a file with the same name already exists, the download exits successfully without doing anything. If you wish to re-download the archive, delete or rename the existing one. == unpack == The ''unpack'' function takes a mandatory and optional argument. The mandatory, first, argument is the name of an archive file (tar.gz, tar.bz2, tar.xz, zip, etc.) to extract. The archive will be unpacked into a directory named ''src'' under the install prefix (versioned directory for installation) unless the optional second argument is specified, then it will be used in place of the name ''src''. This directory, and its parents if necessary, will be created prior to extraction. Source archives using the ''tar'' format customarily have a single top-level directory entry which contains the package name and version, this is automatically stripped from the extracted archive. After completing the extraction process, the ''unpack'' function places the udbuild script into the newly created directory to prepare the script for configure and make steps. If the ''src'' (or alternately specified) directory exists, then the archive is not extracted over it. In this case, the function returns successfully without doing anything. If you wish to force a new extraction, remove or rename the existing ''src'' directory. == create_valet_template == Create a YAML based valet package file template and place it in the ''attic'' directory if one can be found, and the same directory as the udbuild script if one cannot. This template is helpful, but usually cannot just be copied into place. For example, it only knows about the version of the software it is installing, and copying the file blindly would remove entries for all other versions. Furthermore, the "dependencies" entry is filled with all loaded valet packages, even if they are dependencies of dependencies, and not needed to be explicitly listed. == valet == This function takes either the name of a package (e.g. ''openmpi''), or a package name/version pair (e.g. ''openmpi/1.8.2'') and return true if there is a VALET package loaded to satisfy this dependency, and false otherwise. This function can be used along with any other shell constructs, such as ''if'' ... ''else'' ... ''fi'', to modify the behaviour of a build. == version == This function takes a string and validates that it exists as a complete entry (i.e. starts, stops, or is bounded by hyphens) in the VERSION string. This function can be used along with any other shell constructs, such as ''if'' ... ''else'' ... ''fi'', to modify the behavior of a build. == package == This function takes a string and validates that it exists as part of the final package name, which may include features. This is useful for matching features which are specified to configure a software build but don't show up in the version string or require a valet package be available, the string "threads" for OpenMP is a good example of using this feature. == ifvalet == This function is shorthand for ''if valet "$1"; then shift; eval "$@"; fi'' to make udbuild scripts simple to read and code. == ifversion == This function is shorthand for ''if version "$1"; then shift; eval "$@"; fi'' to make udbuild scripts simple to read and code. == ifpackage == This function is shorthand for ''if package "$1"; then shift; eval "$@"; fi'' to make udbuild scripts simple to read and code. == udbuildcapture == Put all screen output into a capture file. The main purpose of this is to log questions answered during an interactive isntall, to document what choices were made. == udbuildmon == This script is helpful to be run during the install phase of a build, for example: udbuildmon make install It will log all ''open'' for write and ''mkdir'' system calls and log them to a file named ''udbuildmon.log''. You can use this log file to verify the build did not write any files to unknown locations. This function should not be necessary with [[http://www.cmake.org|cmake]] builds, as they normally store this information in an ''install_manifest.txt'' file. == apath == Append a path to a variable and ensure that variable is exported. The required first argument is the environment variable name, all remaining arguments are paths to append to the end of the environment variable. A colon (:) character is used as the delimiter, as is standard in path environment variables. == ppath == Prepend a path to a variable similar to ''apath'', but instead of adding the path to the end, add it to the beginning. Arguments are the same as ''apath'' == rpath == Remove a path from an environment variable. The required first argument is the environment variable name. All remaining arguments are removed from the environment variable. If an entry exists multiple times, all instances are removed. == aflag == Append a flag to an environment variable. The required first argument is the environment variable name. All remaining arguments are added to the environment variable. The ''aflag'' variable works under two contexts, which depend on the status of the first argument. If it is an already defined bash array variable type, then the remaining arguments are added as new elements in the array. In all other cases, the remaining arguments are added to the string using a space character as a delimiter. Using a bash array has the advantage of allowing flags which contain whitespace characters. If this is a requirement, the following steps should be undertaken: CONFIG=() #Declare CONFIG as a bash array aflag CONFIG --prefix="$PREFIX" #Specify flags, this one is regular aflag CONFIG --title="My Software" #Specify flags, his one with spaces ./configure "${CONFIG[@]}" #bash syntax for arrays as arguments == pflag == Prepend flags to an environment variable. This is the same as the ''aflag'' function, but it puts its arguments at the beginning of the variable. Its arguments are identical. == rflag == Remove a flag from an environment variable. This works similar to the ''rpath'' variable and also supports bash arrays. == udexpect == This is a wrapper around the TCL ''expect'' utility to simplify the process of answering questions for interactive builds. This function accepts an expect script as STDIN (the normal method is via HERE-DOC) and provides all the basics of running ''expect''. Some standard responses are provided to simplify the process: - enter - Send a carrige return as if the user pressed their "Enter" key - yes - Send the string "yes" as if the user typed "yes" and pressed "Enter" - no - Send "no" and press "Enter" - y - Send "y" and press "Enter" - n - Send "n" and press "Enter" - respond text - Send //text// and press "Enter" - keypress c - Send the character //c// and DO NOT press "Enter" - user - Prompt the person at the keyboard for a respone, and send it, press "Enter" == makeflags_set == Update a file (presumably a Makefile and specified as the first argument) which uses the syntax "key=value" and update the value of the second argument to be that of the third argument. This is a simple helper function to make it simple to edit basic information in a Makefile. == makeflags_prepend == Update a file similar to ''makeflags_set'', except prepend the third argument to the existing value, instead of replacing it. == makeflags_append == Update a file similar to ''makeflags_set'', except append the third argument to the existing value, instead of replacing it. === udbuild script examples === == simple == In this example, an easy-to-install software package called cmake is built and isntalled. It has no software dependencies, and uses the standard ''configure'', ''make'', ''make install'' procedure used by very many open source software packages. To prepare for this build, you would want to create the following directories: <file sh udbuild> #/bin/bash -l PKGNAME=cmake #These are required variables and must VERSION=3.11.3 #be set before calling 'init_udbuildenv' #Setting the "SITEURL" is not required, but is #helpful later. SITEURL=https://cmake.org/files/v${VERSION%.*} PKGINFO='Cross-Platform Make' #Helpful for the ''create_valet_template'' function, URLINFO=http://www.cmake.org #but not required at all vpkg_devrequire udbuild/2 #Use VALET to load the udbuild v2 environment init_udbuildenv #Initialize the udbuild environment #Download the source file if it doesn't already exist download $SITEURL/$PKGNAME-$VERSION.tar.gz unpack $PKGNAME-$VERSION.tar.gz #Unpack the source file into $PREFIX/src and cd there create_valet_template #Create a template file for valet in the attic (you #can't just copy this into place) ./configure --prefix=$PREFIX #Run your normal configure, without #having to define your own PREFIX #variable, because 'init_udbuildenv' did #that for you. make #normal make commands udbuildmon make install #wrap your 'make install' with the #'udbuildmon' function to log what files #and directories were changed. </file> It is imperitive to start udbuild scripts with the string ''#!/bin/bash -l'' because this instructs bash to setup the VALET system. == medium == <file sh udbuild> #!/bin/bash -l PKGNAME=cdo VERSION=1.9.4 SITEURL=https://code.mpimet.mpg.de/attachments/download/17374/ vpkg_devrequire udbuild/2 eccodes/2.8.0:threads proj/5.1.0 vpkg_devrequire netcdf/4.6.1 udunits/2.2.26 fftw/3.3.8 init_udbuildenv create_valet_template download $SITEURL/$PKGNAME-$VERSION.tar.gz unpack $PKGNAME-$VERSION.tar.gz aflag CONFIG --with-szlib="$SZIP_PREFIX" aflag CONFIG --with-hdf5="$HDF5_PREFIX" aflag CONFIG --with-netcdf="$NETCDF_PREFIX" aflag CONFIG --with-eccodes="$ECCODES_PREFIX" aflag CONFIG --with-proj="$PROJ_PREFIX" aflag CONFIG --with-udunits2="$UDUNITS_PREFIX" aflag CONFIG --with-threads=yes aflag CONFIG --with-curl=yes aflag CONFIG --with-libxml=yes ./configure --prefix="$PREFIX" $CONFIG sed -i 's/#include <string>/#include <string.h>/' src/modules.cc make udbuildmon make install </file> In this example, we use ''vpkg_devrequire'' to specify additional dependencies needed to build the ''cdo'' package. ''PREFIX'', however, will still be set to ''/opt/shared/cdo/1.6.4''. == complex == <file sh udbuild> #!/bin/bash -l PKGNAME=hdf4 VERSION=4.2.13 SITEURL=https://support.hdfgroup.org/ftp/HDF/HDF_Current/src PKGINFO='HDF4: Data Model, Library, & File Format' URLINFO=http://www.hdfgroup.org/products/hdf4/ vpkg_devrequire udbuild szip/2.1.1 init_udbuildenv create_valet_template download $SITEURL/hdf-$VERSION.tar.bz2 unpack hdf-$VERSION.tar.bz2 aflag CFLAGS -fPIC if valet intel; then aflag CFLAGS -qopt-jump-tables=large fi aflag CONFIG --disable-netcdf aflag CONFIG --with-szlib=$SZIP_PREFIX # Make shared libraries (sans fortran support): ./configure --prefix="$PREFIX" --enable-shared --disable-fortran $CONFIG make install make clean # Make fortran enabled HDF4: ./configure --prefix="$PREFIX" --disable-shared --enable-fortran $CONFIG make install </file> In this more complicated example, we still need dependencies, but this time one of them will affect the ''PREFIX'' variable. The Intel64 compiler will be used, and PREFIX will be set to ''/opt/shared/hdf4/4.2.10-intel64''. Furthermore, specific ''CFLAGS'' changes will be made for this compiler. This example also illustrates how the ''VERSION'' string can be used. Here, we would set additional flags for the ''./configure'' script if the ''VERSION'' string were set to ''4.2.10-sansnetcdf''. These options allow one build file to build multiple versions of a package, and with only minor changes near the top of the script (namely to the ''VERSION'' variable and the ''init_udbuildenv'' command-line. Another interesting thing we do here is to make sure the installation is as complete as possible. HDF4 does not support shared object files for fortran libraries. So, first we build the shared objects which are possible, then we enable fortran and ensure the full compliment of archive ''.a'' files are present. == python == <file sh udbuild-cdf> #!/bin/bash -l PKGNAME=cdf VERSION=20180613 PY_VER=3.6.5 NETCDF_VER=4.6.1 PKGINFO='Python NetCDF, HDF5, HDF4, and dependent Modules' URLINFO=https://pypi.org/ vpkg_devrequire udbuild python/$PY_VER netcdf/$NETCDF_VER init_udbuildenv python-addon create_valet_template ppath PATH "$PREFIX/bin" apath LD_RUN_PATH "$PREFIX/lib" pip_install CDF cdflib netCDF4 h5netCDF wera2netcdf Puppy nco pip_install python-hdf4 hdf5able hdf5pickle hdf5storage pip_install dtt2hdf ascii2hdf5 h5json h5pyd hdf5_matlab_reader pip_install HDFconvert mrr2c hdfdict LazyHDF5 simpletraj mdtraj pip_install mriqc ncagg pip_install 'MDAnalysis[analysis, AMBER]' </file> The python example above is used to install the cdf and related modules as an add-on for python version 3.6.5. It is also considered a complex example since it displays the use of the option ''python-addon'' for ''init_udbuildenv'' to initialize the udbuild environment and ensure a python VALET package is loaded, sets ''PREFIX'' appropriately for python version's add-on installation path, and utilizes the udbuild ''pip_install'' function to call pip with using the correct ''PREFIX''. To use the python add-on module bundle, you can setup a VALET package. Below is an example VALET yaml package created from the ''create_valet_template'' function. It should be trimmed, as szip, hdf4, and hdf5 are all dependencies of netcdf. The dependencies section should look like the ''vpkg_devrequire'' calls of the udbuild script. <file sh python-cdf.vpkg_yaml> python-cdf: description: Python NetCDF, HDF5, HDF4, and dependent Modules url: https://pypi.org/ prefix: /opt/shared/python/add-ons default-version: "3.6.5:20180613" versions: "3.6.5:20180613": description: cdf compiled with system compilers prefix: python3.6.5/cdf/20180613 dependencies: - python/3.6.5 - szip/2.1.1 - hdf4/4.2.13 - hdf5/1.10.2 - netcdf/4.6.1 </file> abstract/darwin/install_software/udbuild.txt Last modified: 2021-04-22 11:37by anita