Differences

This shows you the differences between two versions of the page.

--- abstract:caviness:install_software:install_software [2021-02-22 15:46] – [UDBUILD Software Deployment] anita
+++ abstract:caviness:install_software:install_software [2024-02-12 11:14] (current) – anita
@@ Line 1: / Line 1: @@
-===== Software Installation =====
+===== Software Installation on Caviness =====
-The HPC team has a set of standards and technology used to reduce complexity and bring
-consistency to the process.
-Software is generally built, installed, and accessed using the
+First see if your software is installed by using ''vpkg_list'' on Caviness, and then check the [[software:software|Software]] documentation for all software which is organized in alphabetical order on the <html><span style="color:#ffffff;background-color:#2fa4e7;padding:3px 7px !important;border-radius:4px;">sidebar</span></html> to see if there are any specific instructions provided for use and/or installation on Caviness.
-[[software:valet:valet|VALET]] system developed by Dr. Jeffrey Frey.
-==== Filesystem ====
+The HPC team has a set of standards and technology used to reduce complexity and bring consistency to the process of software installation and management.
-Software is deployed to ''/opt/shared''.
-The ''udbuild'' system defaults to this ''/opt/shared''
-location.  However, this can be changed by setting the ''UDBUILD_HOME''
-environment variable before initializing the ubuild environment.  A good
-value for this environment variable for workgroup software is
-''$WORKDIR/sw'', and for personal software installation is
-''$HOME/sw''. Refer to [[:abstract:caviness:.install_software:workgroup-sw|workgroup software installs]] for help setting up your directories for workgroup storage.
-Beneath this directory should be an ''attic'' sub-directory for downloaded
+Software is generally built, installed, and accessed using the [[software:valet:valet|VALET]] system (not Modules but similar) developed by Dr. Jeffrey Frey. VALET provides the ability to modify your environment **without** modifying your startup files like ''.bashrc'' and ''.bash_profile'' as it prevents jobs from failing by keeping a clean login environment.
-software bundles, optionally an ''add-ons'' directory for software with
-optional add-ons, and one sub-directory for each package installed.
-These sub-directories should always be in all lower-case letter.  One more
-layer down should be a directory for each version of the software installed.
-It is important understand that on a complex cluster like Caviness, the same
-release of a software package may have multiple installations due to various
-compiler and dependency package requirements.  These directories are the
-software installation roots.
-Underneath the installation root should be a directory called ''src'', which
+This process of basic software management and installation is described in [[technical:recipes:software-managment|Software Management]] and [[abstract:caviness:install_software:workgroup-sw|Workgroup Directory]], where both can also be found on the <html><span style="color:#ffffff;background-color:#2fa4e7;padding:3px 7px !important;border-radius:4px;">sidebar</span></html> under [[abstract:caviness:install_software:install_software|Install Software]].
-is the un-packed source bundle.  Next to ''src'' should be any ''bin'',
-''lib'', ''share'', etc. directories neccessary for the final deployment.
-An illustrated example of the software directory structure is as such:
+With that said, you may need to use other methodologies for installing your software such as a [[software:python:python|Python]] virtualenv or a container technology like [[software:singularity:singularity|Singularity]] on the cluster along with creating a VALET package to properly add it to your environment.
-  * opt
-    * shared
-      * atlas
-         * 3.10.3
-         * 3.10.3-intel
-         * attic
-           * ''udbuild''  <sup>// - build and install script for atlas//</sup>
-      * python
-         * 2.7.8
-         * 3.2.5
-         * add-ons
-           * python2.7.15
-             * mpi
-               * 20180613
-           * python3.2.5
-         * attic
-           * ''udbuild''   <sup>// - build and install script for python//</sup>
-==== Building ====
-When building software, the base directory structure (including the ''attic''
-directory) should be created by you before proceeding further.
-You should download the software source bundle into ''attic''.  Then,
-unpack the software bundle and rename the directory to ''src'' as
-above.  This provides consistency in finding the source bundle and the
-''udbuild'' file.
-Examples of builds are provided below (after the udbuild function
-documentation).
-=== udbuild functions ===
-== init_udbuildenv ==
-This function initializes the udbuild environment. It ensures that you have
-the required ''PKGNAME'' and ''VERSION'' environment variables defined, you do
-not have VALET packages loaded before ''udbuild'' in your VALET history (these
-might affect your build), sets compiler variables like CC, FC, etc., then
-finally sets your ''PREFIX'' and ''VERSION'' variables based on it's
-command-line.  These command-line options affect ''init_udbuildenv'':
-  * none - This is equivalent to not supplying any parameters
-  * python-addon - Ensure a python VALET package is loaded, and set PREFIX \
-    appropriately for that python version's add-on installation path
-  * r-addon - Ensure an R VALET package is loaded, and set PREFIX \
-    appropriately for that R version's add-on installation path
-  * node-addon - Ensure a Node.JS version is loaded and set the PREFIX \
-    appropriately for that Node.JS versions' add-on installation path
-  * Any other arguments are treated as the names of VALET packages which are \
-    loaded and added to the ''VERSION'' environment variable.
-After all of this, your ''PREFIX'' variable will be set to
-  ${UDBUILD_HOME:-/opt/shared}/$PKGNAME/$VERSION
-== debug ==
-Drop into a debug shell.  If the debug shell is exited cleanly, then the
-udbuild script will continue from there.  This is a useful routine to use
-when creating a udbuild script.  You may want to build the script based
-on documentation, and run a ''debug'' function between a configure and
-make step to verify the environment looks sane.  After the first successful
-compile, you can then remove the debug line.
-== download ==
-The ''download'' function takes a mandatory and optional argument.  The mandatory,
-first, argument is the URL to download.  The resulting file will be named after
-the last part of the URL unless the optional second argument is specified.  In this
-case, the resulting file will be named after the second argument.
-If a file with the same name already exists, the download exits successfully without
-doing anything.  If you wish to re-download the archive, delete or rename the existing
-one.
-== unpack ==
-The ''unpack'' function takes a mandatory and optional argument.  The mandatory,
-first, argument is the name of an archive file (tar.gz, tar.bz2, tar.xz, zip, etc.)
-to extract.  The archive will be unpacked into a directory named ''src'' under the
-install prefix (versioned directory for installation) unless the optional second argument
-is specified, then it will be used in place of the name ''src''.  This directory, and
-its parents if necessary, will be created prior to extraction.  Source archives using
-the ''tar'' format customarily have a single top-level directory entry which contains
-the package name and version, this is automatically stripped from the extracted archive.
-After completing the extraction process, the ''unpack'' function places the udbuild
-script into the newly created directory to prepare the script for configure and make
-steps.
-If the ''src'' (or alternately specified) directory exists, then the archive is not
-extracted over it.  In this case, the function returns successfully without doing
-anything.  If you wish to force a new extraction, remove or rename the existing
-''src'' directory.
-== create_valet_template ==
-Create a YAML based valet package file template and place it in the ''attic'' directory
-if one can be found, and the same directory as the udbuild script if one cannot.  This
-template is helpful, but usually cannot just be copied into place.  For example, it only
-knows about the version of the software it is installing, and copying the file blindly would
-remove entries for all other versions.  Furthermore, the "dependencies" entry is filled with
-all loaded valet packages, even if they are dependencies of dependencies, and not needed
-to be explicitly listed.
-== valet ==
-This function takes either the name of a package (e.g. ''openmpi''), or a
-package name/version pair (e.g. ''openmpi/1.8.2'') and return true if there
-is a VALET package loaded to satisfy this dependency, and false otherwise.
-This function can be used along with any other shell constructs, such as
-''if'' ... ''else'' ... ''fi'', to modify the behaviour of a build.
-== version ==
-This function takes a string and validates that it exists as a complete
-entry (i.e. starts, stops, or is bounded by hyphens) in the VERSION string.
-This function can be used along with any other shell constructs, such as
-''if'' ... ''else'' ... ''fi'', to modify the behavior of a build.
-== package ==
-This function takes a string and validates that it exists as part of
-the final package name, which may include features.  This is useful for
-matching features which are specified to configure a software build but
-don't show up in the version string or require a valet package be
-available, the string "threads" for OpenMP is a good example of using
-this feature.
-== ifvalet ==
-This function is shorthand for ''if valet "$1"; then shift; eval "$@"; fi''
-to make udbuild scripts simple to read and code.
-== ifversion ==
-This function is shorthand for ''if version "$1"; then shift; eval "$@"; fi''
-to make udbuild scripts simple to read and code.
-== ifpackage ==
-This function is shorthand for ''if package "$1"; then shift; eval "$@"; fi''
-to make udbuild scripts simple to read and code.
-== udbuildcapture ==
-Put all screen output into a capture file.  The main purpose of this is to
-log questions answered during an interactive isntall, to document what
-choices were made.
-== udbuildmon ==
-This script is helpful to be run during the install phase of a build, for
-example:
-  udbuildmon make install
-It will log all ''open'' for write and ''mkdir'' system calls and log them to
-a file named ''udbuildmon.log''.  You can use this log file to verify the
-build did not write any files to unknown locations.  This function should not
-be necessary with [[http://www.cmake.org|cmake]] builds, as they normally
-store this information in an ''install_manifest.txt'' file.
-== apath ==
-Append a path to a variable and ensure that variable is exported.  The required
-first argument is the environment variable name, all remaining arguments are
-paths to append to the end of the environment variable.  A colon (:) character
-is used as the delimiter, as is standard in path environment variables.
-== ppath ==
-Prepend a path to a variable similar to ''apath'', but instead of adding the
-path to the end, add it to the beginning.  Arguments are the same as ''apath''
-== rpath ==
-Remove a path from an environment variable.  The required first argument is the
-environment variable name.  All remaining arguments are removed from the
-environment variable.  If an entry exists multiple times, all instances are
-removed.
-== aflag ==
-Append a flag to an environment variable.  The required first argument is the
-environment variable name.  All remaining arguments are added to the environment
-variable.  The ''aflag'' variable works under two contexts, which depend on the
-status of the first argument.  If it is an already defined bash array variable
-type, then the remaining arguments are added as new elements in the array.  In
-all other cases, the remaining arguments are added to the string using a space
-character as a delimiter.
-Using a bash array has the advantage of allowing flags which contain whitespace
-characters.  If this is a requirement, the following steps should be undertaken:
-  CONFIG=()                            #Declare CONFIG as a bash array
-  aflag CONFIG --prefix="$PREFIX"      #Specify flags, this one is regular
-  aflag CONFIG --title="My Software"   #Specify flags, his one with spaces
-  ./configure  "${CONFIG[@]}"          #bash syntax for arrays as arguments
-== pflag ==
-Prepend flags to an environment variable.  This is the same as the ''aflag''
-function, but it puts its arguments at the beginning of the variable.  Its
-arguments are identical.
-== rflag ==
-Remove a flag from an environment variable.  This works similar to the
-''rpath'' variable and also supports bash arrays.
-== udexpect ==
-This is a wrapper around the TCL ''expect'' utility to simplify the process of
-answering questions for interactive builds.  This function accepts an expect
-script as STDIN (the normal method is via HERE-DOC) and provides all the basics
-of running ''expect''.  Some standard responses are provided to simplify the
-process:
-  - enter - Send a carrige return as if the user pressed their "Enter" key
-  - yes - Send the string "yes" as if the user typed "yes" and pressed "Enter"
-  - no - Send "no" and press "Enter"
-  - y - Send "y" and press "Enter"
-  - n - Send "n" and press "Enter"
-  - respond text - Send //text// and press "Enter"
-  - keypress c - Send the character //c// and DO NOT press "Enter"
-  - user - Prompt the person at the keyboard for a respone, and send it, press "Enter"
-== makeflags_set ==
-Update a file (presumably a Makefile and specified as the first argument) which uses
-the syntax "key=value" and update the value of the second argument to be that of the
-third argument.  This is a simple helper function to make it simple to edit basic
-information in a Makefile.
-== makeflags_prepend ==
-Update a file similar to ''makeflags_set'', except prepend the third argument to the
-existing value, instead of replacing it.
-== makeflags_append ==
-Update a file similar to ''makeflags_set'', except append the third argument to the
-existing value, instead of replacing it.
-=== udbuild script examples ===
-== simple ==
-In this example, an easy-to-install software package called cmake is built
-and isntalled.  It has no software dependencies, and uses the standard
-''configure'', ''make'', ''make install'' procedure used by very many
-open source software packages.
-To prepare for this build, you would want to create the following directories:
-<file sh udbuild>
-#/bin/bash -l
-PKGNAME=cmake                       #These are required variables and must
-VERSION=3.11.3                      #be set before calling 'init_udbuildenv'
-                                    #Setting the "SITEURL" is not required, but is
-                                    #helpful later.
-SITEURL=https://cmake.org/files/v${VERSION%.*}
-PKGINFO='Cross-Platform Make'       #Helpful for the ''create_valet_template'' function,
-URLINFO=http://www.cmake.org        #but not required at all
-vpkg_devrequire udbuild/2           #Use VALET to load the udbuild v2 environment
-init_udbuildenv                     #Initialize the udbuild environment
-                                    #Download the source file if it doesn't already exist
-download $SITEURL/$PKGNAME-$VERSION.tar.gz
-unpack $PKGNAME-$VERSION.tar.gz     #Unpack the source file into $PREFIX/src and cd there
-create_valet_template               #Create a template file for valet in the attic (you
-                                    #can't just copy this into place)
-./configure --prefix=$PREFIX        #Run your normal configure, without
-                                    #having to define your own PREFIX
-                                    #variable, because 'init_udbuildenv' did
-                                    #that for you.
-make                                #normal make commands
-udbuildmon make install             #wrap your 'make install' with the
-                                    #'udbuildmon' function to log what files
-                                    #and directories were changed.
-</file>
-It is imperitive to start udbuild scripts with the string ''#!/bin/bash -l''
-because this instructs bash to setup the VALET system.
-== medium ==
-<file sh udbuild>
-#!/bin/bash -l
-PKGNAME=cdo
-VERSION=1.9.4
-SITEURL=https://code.mpimet.mpg.de/attachments/download/17374/
-vpkg_devrequire udbuild/2 eccodes/2.8.0:threads proj/5.1.0
-vpkg_devrequire netcdf/4.6.1 udunits/2.2.26 fftw/3.3.8
-init_udbuildenv
-create_valet_template
-download $SITEURL/$PKGNAME-$VERSION.tar.gz
-unpack $PKGNAME-$VERSION.tar.gz
-aflag CONFIG --with-szlib="$SZIP_PREFIX"
-aflag CONFIG --with-hdf5="$HDF5_PREFIX"
-aflag CONFIG --with-netcdf="$NETCDF_PREFIX"
-aflag CONFIG --with-eccodes="$ECCODES_PREFIX"
-aflag CONFIG --with-proj="$PROJ_PREFIX"
-aflag CONFIG --with-udunits2="$UDUNITS_PREFIX"
-aflag CONFIG --with-threads=yes
-aflag CONFIG --with-curl=yes
-aflag CONFIG --with-libxml=yes
-./configure --prefix="$PREFIX" $CONFIG
-sed -i 's/#include <string>/#include <string.h>/' src/modules.cc
-make
-udbuildmon make install
-</file>
-In this example, we use ''vpkg_devrequire'' to specify additional dependencies
-needed to build the ''cdo'' package.  ''PREFIX'', however, will still be set
-to ''/opt/shared/cdo/1.6.4''.
-== complex ==
-<file sh udbuild>
-#!/bin/bash -l
-PKGNAME=hdf4
-VERSION=4.2.13
-SITEURL=https://support.hdfgroup.org/ftp/HDF/HDF_Current/src
-PKGINFO='HDF4: Data Model, Library, & File Format'
-URLINFO=http://www.hdfgroup.org/products/hdf4/
-vpkg_devrequire udbuild szip/2.1.1
-init_udbuildenv
-create_valet_template
-download $SITEURL/hdf-$VERSION.tar.bz2
-unpack hdf-$VERSION.tar.bz2
-aflag CFLAGS -fPIC
-if valet intel; then
-        aflag CFLAGS -qopt-jump-tables=large
-fi
-aflag CONFIG --disable-netcdf
-aflag CONFIG --with-szlib=$SZIP_PREFIX
-# Make shared libraries (sans fortran support):
-./configure --prefix="$PREFIX" --enable-shared --disable-fortran $CONFIG
-make install
-make clean
-# Make fortran enabled HDF4:
-./configure --prefix="$PREFIX" --disable-shared --enable-fortran $CONFIG
-make install
-</file>
-In this more complicated example, we still need dependencies, but this time
-one of them will affect the ''PREFIX'' variable.  The Intel64 compiler will
-be used, and PREFIX will be set to ''/opt/shared/hdf4/4.2.10-intel64''.
-Furthermore, specific ''CFLAGS'' changes will be made for this compiler.
-This example also illustrates how the ''VERSION'' string can be used.  Here,
-we would set additional flags for the ''./configure'' script if the ''VERSION''
-string were set to ''4.2.10-sansnetcdf''.  These options allow one build file
-to build multiple versions of a package, and with only minor changes near
-the top of the script (namely to the ''VERSION'' variable and the
-''init_udbuildenv'' command-line.
-Another interesting thing we do here is to make sure the installation is
-as complete as possible.  HDF4 does not support shared object files for
-fortran libraries.  So, first we build the shared objects which are possible,
-then we enable fortran and ensure the full compliment of archive ''.a'' files
-are present.
-== python ==
-<file sh udbuild-cdf>
-#!/bin/bash -l
-PKGNAME=cdf
-VERSION=20180613
-PY_VER=3.6.5
-NETCDF_VER=4.6.1
-PKGINFO='Python NetCDF, HDF5, HDF4, and dependent Modules'
-URLINFO=https://pypi.org/
-vpkg_devrequire udbuild python/$PY_VER netcdf/$NETCDF_VER
-init_udbuildenv python-addon
-create_valet_template
-ppath PATH        "$PREFIX/bin"
-apath LD_RUN_PATH "$PREFIX/lib"
-pip_install CDF cdflib netCDF4 h5netCDF wera2netcdf Puppy nco
-pip_install python-hdf4 hdf5able hdf5pickle hdf5storage
-pip_install dtt2hdf ascii2hdf5 h5json h5pyd hdf5_matlab_reader
-pip_install HDFconvert mrr2c hdfdict LazyHDF5 simpletraj mdtraj
-pip_install mriqc ncagg
-pip_install 'MDAnalysis[analysis, AMBER]'
-</file>
-The python example above is used to install the cdf and related modules as an add-on for python version 3.6.5. It is also considered a complex example since it displays the use of the option ''python-addon'' for ''init_udbuildenv'' to initialize the udbuild environment and ensure a python VALET package is loaded, sets ''PREFIX'' appropriately for python version's add-on installation path, and utilizes the udbuild ''pip_install'' function to call pip with using the correct ''PREFIX''. To use the python add-on module bundle, you can setup a VALET package.  Below is an example VALET yaml package created from the ''create_valet_template'' function.  It should be trimmed, as szip, hdf4, and hdf5 are all dependencies of netcdf.  The dependencies section should look like the ''vpkg_devrequire'' calls of the udbuild script.
-<file sh python-cdf.vpkg_yaml>
-python-cdf:
-  description:          Python NetCDF, HDF5, HDF4, and dependent Modules
-  url:                  https://pypi.org/
-  prefix:               /opt/shared/python/add-ons
-  default-version:      "3.6.5:20180613"
-  versions:
-    "3.6.5:20180613":
-      description:      cdf compiled with system compilers
-      prefix:           python3.6.5/cdf/20180613
-      dependencies:
-        -               python/3.6.5
-        -               szip/2.1.1
-        -               hdf4/4.2.13
-        -               hdf5/1.10.2
-        -               netcdf/4.6.1
-</file>