====== Software Management ====== This document offers an introduction to the mechanisms — organizational and procedural — used by IT RCI staff to build and manage the software made available to users on the HPC systems: === Organizational === It is most often the case that you do not use a single version of a piece of software. Bug fixes and feature additions drive change in software over time, leading to //versioned releases// of the product. In many cases the same versioned release may have mutually exclusive features that require multiple //variants// of that version to be maintained (simplest example: differing compiler toolchains, GCC versus Intel). Organization is also important in maintaining //reproducibility// of your work. An unstable, difficult to maintain computing platform is the product of: * Maintaining a full copy of your source code with each calculation: minor to massive amounts of storage space can be wasted, and keeping multiple source trees in-sync is a challenge * Copying executables and libraries into each working directory: difficult to remain consistent on release version, feature set, build parameters, etc. involved in producing the executable * Including extensive Unix environment manipulation in every job script: any correction to the environment manipulation must be applied to every affected job script to remain consistent === Procedural === For many versions and variants of software to coexist on a system and be used properly, users cannot just modify their login files (e.g. ''.bashrc'') to alter environment variables: such change affects //every shell// the user subsequently launches. Environment management tools like VALET encapsulate the changes so they can be made consistently and easily. Corrections can be made to the encapsulated change information to affect globally (for that version or variant of the software). VALET also makes it easier to build software. Standard development paths — like the ''/lib'' and ''/include'' directories — are automatically added to key environment variables when the //development context// is used: [user@login00.darwin ~]$ vpkg_devrequire udunits/2.2.28 Adding package `udunits/2.2.28` to your environment [user@login00.darwin ~]$ echo $UDUNITS_PREFIX /opt/shared/udunits/2.2.28 [user@login00.darwin ~]$ echo $LDFLAGS -L/opt/shared/udunits/2.2.28/lib [user@login00.darwin ~]$ echo $CPPFLAGS -I/opt/shared/udunits/2.2.28/include The GNU Autoconf build system makes use of these environment variables when searching for and building software, and other tools can make use of the value of the variable (e.g. ''cmake -DUDUNITS_ROOT=$UDUNITS_PREFIX''). VALET is not just a tool that system administrators can use to describe environment changes: all users can create their own //package definitions// for themselves (in their ''~/.valet'' directory) or for their workgroup (in the ''$WORKDIR/sw/valet'' directory). A package definition can reference other packages as //dependencies// and have those dependencies automatically loaded into the environment. ===== Preparations ===== In this example we will build the latest version of the [[https://tddft.org/programs/libxc/|libxc]] library using various compilers. First and foremost, decide where on the file system you are going to organize the set of versions/variants of the software. For a user maintaining personal builds of the software the home directory can be used: [user@login00.darwin ~]$ LIBXC_BASEDIR=~/sw/libxc For shared installations made available to your entire workgroup: On DARWIN: [(workgroup:user)@login00.darwin ~]$ LIBXC_BASEDIR="${WORKDIR_SW}/libxc" On Caviness: [(workgroup:user)@login00 ~]$ LIBXC_BASEDIR="${WORKDIR}/sw/libxc" Ensure the base directory exists: [user@login00.darwin ~]$ mkdir -p "$LIBXC_BASEDIR" Often the software being built is distributed as one or more downloadable files (e.g. tar.gz archive). IT RCI staff create a directory to hold (organize!) these files. For **libxc** we will create that directory and download the source archive: [user@login00.darwin ~]$ mkdir -p "$LIBXC_BASEDIR/attic" [user@login00.darwin ~]$ wget -O "$LIBXC_BASEDIR/attic/libxc-5.1.0.tar-gz" \ "http://www.tddft.org/programs/libxc/down.php?file=5.1.0/libxc-5.1.0.tar.gz" At this point the source code has been downloaded, our directory hierarchy has been established, and we are ready to build a variant of the 5.1.0 release. ===== Builds ===== Each of the compilers chosen in building //variants// of the 5.1.0 release will follow the same general procedure. The notation '':'' (colon) in output examples throughout this document indicates lines of output are displayed by the command entered but have been omitted from this documentation. ==== System GCC ==== The cluster comes with a native GNU toolchain present (gcc, g++, gfortran). No environment changes are necessary to use these compilers. Since it is the default compiler toolchain, IT RCI staff do not mention it when naming a software variant: this variant will have a //version identifier// of ''5.1.0''. Prepare the variant's installation directory and unpack the source code inside it: [user@login00.darwin ~]$ mkdir "$LIBXC_BASEDIR/5.1.0" [user@login00.darwin ~]$ cd "$LIBXC_BASEDIR/5.1.0" [user@login00.darwin 5.1.0]$ tar -xf "$LIBXC_BASEDIR/attic/libxc-5.1.0.tar-gz" [user@login00.darwin 5.1.0]$ mv libxc-5.1.0 src [user@login00.darwin 5.1.0]$ cd src The **libxc** build system uses GNU Autoconf or CMake. For this variant, we will use Autoconf: [user@login00.darwin src]$ ./configure --prefix="$LIBXC_BASEDIR/5.1.0" Additional flags may be necessary, but the most important is the ''--prefix'', telling the build system the software is intended to be installed into the chosen directory for the variant. [user@login00.darwin src]$ make : xc-threshold.c: In function ‘check_xc’: xc-threshold.c:808:3: error: ‘for’ loop initial declarations are only allowed in C99 mode for (int i = 0; i < (int) (sizeof(xc_values_type) / sizeof(double)); i++) ^ xc-threshold.c:808:3: note: use option -std=c99 or -std=gnu99 to compile your code This build encountered an error: the system GNU C compiler defaults to an older language standard than what the source code implies, so the compiler has mentioned that a flag is necessary to choose that standard: [user@login00.darwin src]$ CFLAGS="-std=gnu99" ./configure --prefix="$LIBXC_BASEDIR/5.1.0" : [user@login00.darwin src]$ make : [user@login00.darwin src]$ make install : ==== Intel ==== The Intel compilers offer advanced optimizations to target the specific processor models' features. To use the Intel compilers they must be first added to the environment: [user@login00.darwin ~]$ cd [user@login00.darwin ~]$ vpkg_require intel/2020u4 Adding package `intel/2020u4` to your environment [user@login00.darwin ~]$ which icc /opt/shared/intel/2020u4/compilers_and_libraries_2020.4.304/linux/bin/intel64/icc We will also use CMake for this build; though the OS does provide a version of CMake, it is relatively old so a newer version is often necessary: [user@login00.darwin ~]$ vpkg_require cmake/3.19.1 Adding package `cmake/3.19.1` to your environment Since this variant of 5.1.0 uses the Intel compilers, it will have a version identifier that includes a //feature// named ''intel-2020''. The appropriate directory name for that identifier can be determined using: [user@login00.darwin ~]$ vpkg_id2path --version="5.1.0:intel-2020" 5.1.0-intel-2020 Leading to the unpack procedure: [user@login00.darwin ~]$ mkdir "$LIBXC_BASEDIR/5.1.0-intel-2020" [user@login00.darwin ~]$ cd "$LIBXC_BASEDIR/5.1.0-intel-2020" [user@login00.darwin 5.1.0-intel-2020]$ tar -xf "$LIBXC_BASEDIR/attic/libxc-5.1.0.tar-gz" [user@login00.darwin 5.1.0-intel-2020]$ mv libxc-5.1.0 src [user@login00.darwin 5.1.0-intel-2020]$ cd src CMake builds usually request that you create an empty directory to hold all of the files generated by CMake. The commands analogous to the Autoconf setup of the build look like: [user@login00.darwin 5.1.0-intel-2020]$ mkdir build [user@login00.darwin 5.1.0-intel-2020]$ cd build [user@login00.darwin build]$ CC=icc FC=ifort CXX=icpc cmake \ -DCMAKE_INSTALL_PREFIX="$LIBXC_BASEDIR/5.1.0-intel-2020" \ -DBUILD_TESTING=FALSE \ .. : The CMake build system doesn't yet support Fortran compilation 100%, so by default that language is omitted (and the ''FC=ifort'' is probably extraneous). By the same token, none of the source is written in C++ so specifying ''CXX=icpc'' is also extraneous, but in neither case does it hurt to have your intentions clear. [user@login00.darwin build]$ make : [user@login00.darwin build]$ make install : ==== GCC 10 ==== Finally, a variant using the GCC 10 compiler will be built using Autoconf. First, remove all environment changes made for the Intel build [user@login00.darwin build]$ cd [user@login00.darwin ~]$ vpkg_rollback all and configure the environment for GCC 10 and the newer CMake: [user@login00.darwin ~]$ vpkg_versions gcc Available versions in package (* = default version): [/opt/shared/valet/2.1/etc/gcc.vpkg_yaml] gcc GCC Compiler Suite 4.8 alias to gcc/4.8.5 4.8.5 CentOS system GCC with C, C++, Obj-C, Obj-C++, and Fortran 7.3 alias to gcc/7.3.0 7.3.0 GCC with C, C++, Obj-C, Obj-C++, Fortran, and GO 10.1.0 GCC with C, C++, Obj-C, Obj-C++, Fortran, and GO 10.1 alias to gcc/10.1.0 * system alias to gcc/4.8.5 [user@login00.darwin ~]$ vpkg_require cmake/3.19.1 gcc/10.1.0 Adding package `cmake/3.19.1` to your environment Adding package `gcc/10.1.0` to your environment Notice that multiple packages can be specified in a single ''vpkg_require'' command — this is more efficient than running them as multiple ''vpkg_require'' commands. As with the Intel example, we will add a ''gcc-10.1'' feature to the version id to indicate the compiler choice, so the preparations look like: [user@login00.darwin ~]$ vpkg_id2path --version="5.1.0:gcc-10.1" 5.1.0-gcc-10.1 [user@login00.darwin ~]$ mkdir "$LIBXC_BASEDIR/5.1.0-gcc-10.1" [user@login00.darwin ~]$ cd "$LIBXC_BASEDIR/5.1.0-gcc-10.1" [user@login00.darwin 5.1.0-gcc-10.1]$ tar -xf "$LIBXC_BASEDIR/attic/libxc-5.1.0.tar-gz" [user@login00.darwin 5.1.0-gcc-10.1]$ mv libxc-5.1.0 src [user@login00.darwin 5.1.0-gcc-10.1]$ cd src Sometimes Autoconf allows the same approach as CMake: creating a build directory to hold the files produced by the build system, leaving the source directory undisturbed. [user@login00.darwin src]$ mkdir build [user@login00.darwin src]$ cd build [user@login00.darwin build]$ CC=gcc FC=gfortran ../configure --prefix="$LIBXC_BASEDIR/5.1.0-gcc-10.1" : [user@login00.darwin build]$ make : [user@login00.darwin build]$ make install : Note that with GCC 10 the additional ''CFLAGS=-std=gnu99'' was not present on the ''../configure'' command. With the 10.1 gcc compiler, the default language standard is listed as **gnu18**. The default for GCC 4.8.5 is **gnu89** — older than the C99 standard the code requires. ==== Results ==== Through the course of the above sections three distinct variants of **libxc** 5.1.0 were produced. The base directory currently looks like: [user@login00.darwin build]$ cd [user@login00.darwin ~]$ vpkg_rollback all [user@login00.darwin ~]$ ls -l "$LIBXC_BASEDIR" total 21 drwxr-xr-x 6 user everyone 6 Feb 8 11:35 5.1.0 drwxr-xr-x 6 user everyone 6 Feb 8 12:11 5.1.0-gcc-10.1 drwxr-xr-x 7 user everyone 7 Feb 8 11:56 5.1.0-intel-2020 drwxr-xr-x 2 user everyone 3 Feb 8 11:10 attic Each variant is structured similarly, with the typical installation directory layout: [user@login00.darwin ~]$ ls -l "$LIBXC_BASEDIR/5.1.0-intel-2020" total 40 drwxr-xr-x 2 user everyone 3 Feb 8 11:56 bin drwxr-xr-x 2 user everyone 7 Feb 8 11:56 include drwxr-xr-x 3 user everyone 4 Feb 8 11:56 lib64 drwxr-xr-x 3 user everyone 3 Feb 8 11:56 share drwxr-xr-x 11 user everyone 40 Feb 8 11:50 src To make use of one of these variants of **libxc**, these directories must be added to specific environment variables: a task for which VALET is designed to assist. ===== VALET Setup ===== Since **libxc** adheres to the standard directory layout for software on Linux (''bin'', ''lib'' or ''lib64'', ''include'') it is relatively easy to setup the runtime environment: * ''$LIBXC_BASEDIR/5.1.0-intel-2020/bin'' must be added to ''$PATH'' * ''$LIBXC_BASEDIR/5.1.0-intel-2020/lib64'' must be added to ''$LD_LIBRARY_PATH'' In addition, for software-building the linker could be told to check ''$LIBXC_BASEDIR/5.1.0-intel-2020/lib64'' for required libraries and the C/C++ compiler told to look in ''$LIBXC_BASEDIR/5.1.0-intel-2020/include'' for header files — more on that in a moment. VALET automatically recognizes the standard directory layout, so configuring these variants of ''libxc'' is very straightforward. First, note where the variants were collocated on the file system: [user@login00.darwin ~]$ echo $LIBXC_BASEDIR /home/user/sw/libxc Since these builds were done in the user's home directory, they were personal copies of the software and should use a //VALET package definition file// stored in ''~/.valet'' [user@login00.darwin ~]$ VALET_PKG_DIR=~/.valet ; VALET_PKG_DIR_MODE=0700 versus an installation made for an entire workgroup would store VALET package definition files in [user@login00.darwin ~]$ VALET_PKG_DIR="$WORKDIR_SW/valet" on DARWIN and in [user@login00.darwin ~]$ VALET_PKG_DIR="$WORKDIR/sw/valet" ; VALET_PKG_DIR_MODE=2770 on Caviness. Whichever scheme is in-use, ensure the directory exists for personal use on Caviness and DARWIN, and entire workgroup on Caviness (entire workgroup on DARWIN is automatically created for each allocation so this is not necessary) [user@login00.darwin ~]$ mkdir -p --mode=$VALET_PKG_DIR_MODE "$VALET_PKG_DIR" VALET allows package definitions in a variety of formats (XML, JSON, YAML) but YAML tends to be the simplest format so we will use it here. ==== Package section ==== The //package section// of the definition file includes items that apply to all versions/variants of the software: libxc: prefix: /home/user/sw/libxc description: a library of exchange-correlation functionals for density-functional theory url: "https://tddft.org/programs/libxc/" The //package identifier// is the top-level key in the document — ''libxc'' — and the value of ''$LIBXC_BASEDIR'' is the value of the ''prefix'' key in this section. The URL and description are information taken from the official ''libxc'' web site. ==== Versions ==== The ''versions'' key is used to provide a list of the versions/variants of the software. The simplest version that was built used the system GCC compiler: libxc: prefix: /home/user/sw/libxc description: a library of exchange-correlation functionals for density-functional theory url: "https://tddft.org/programs/libxc/" versions: "5.1.0": description: compiled with system gcc/gfortran (4.8.5) Since we used ''vpkg_id2path'' to determine the path associated with the version identifiers and each variant is installed under ''$LIBXC_BASEDIR'', there's no need to specify a prefix for the version definitions: package's prefix (''/home/user/sw/libxc'') with the version identifier appended (''/home/user/sw/libxc/5.1.0'') is implicit. The implicit behavior is overridden by providing a ''prefix'' key in the version definition: a relative path is appended to the package's prefix, an absolute path is used as-is. For the other two variants, a dependency exists with respect to the compiler toolchain that was used. Each of those variants should be defined with that dependency described: libxc: prefix: /home/user/sw/libxc description: a library of exchange-correlation functionals for density-functional theory url: "https://tddft.org/programs/libxc/" versions: "5.1.0": description: compiled with system gcc/gfortran (4.8.5) "5.1.0:intel-2020": description: compiled with Intel icc (2020) dependencies: - intel/2020 "5.1.0:gcc-10.1": description: compiled with gcc (10.1) dependencies: - gcc/10.1 We could have been more specific about the version of the Intel and GCC compiler that was used, e.g. ''intel/2020u4'' and ''gcc/10.1.0''. Minor releases of software do not usually significantly change its behavior or alter APIs, so by being less specific (''intel/2020'') if the Intel compiler were upgraded to ''2020u5'' (and the version identifier ''2020'' promoted to point to that by the system administrator) then our ''libxc/5.1.0:intel-2020'' would automatically have the new version of Intel as its dependency. Note that this automatic "upgrade" does not work for static executables and libraries (they would need to be rebuilt to use the new runtime libraries, for example). Finally, it is a good idea to specify which version definition should act as the default. This yields the following package definition file libxc: prefix: /home/user/sw/libxc description: a library of exchange-correlation functionals for density-functional theory url: "https://tddft.org/programs/libxc/" default-version: "5.1.0:gcc-10.1" versions: "5.1.0": description: compiled with system gcc/gfortran (4.8.5) "5.1.0:intel-2020": description: compiled with Intel icc (2020) dependencies: - intel/2020 "5.1.0:gcc-10.1": description: compiled with gcc (10.1) dependencies: - gcc/10.1 saved at ''$VALET_PKG_DIR/libxc.vpkg_yaml''. ==== Checking the definition file ==== The package definition file can be syntax-checked: [user@login00.darwin ~]$ vpkg_check "$VALET_PKG_DIR/libxc.vpkg_yaml" /home/user/.valet/libxc.vpkg_yaml is OK [libxc] { contexts: all actions: { LIBXC_PREFIX=${VALET_PATH_PREFIX} (contexts: development) } https://tddft.org/programs/libxc/ a library of exchange-correlation functionals for density-functional theory prefix: /home/user/sw/libxc source file: /home/user/.valet/libxc.vpkg_yaml default version: libxc/5.1.0:intel-2020 versions: { [libxc/5.1.0] { contexts: all compiled with system gcc/gfortran (4.8.5) prefix: /home/user/sw/libxc/5.1.0 standard paths: { bin: /home/user/sw/libxc/5.1.0/bin lib: /home/user/sw/libxc/5.1.0/lib include: /home/user/sw/libxc/5.1.0/include pkgConfig: /home/user/sw/libxc/5.1.0/lib/pkgconfig } } [libxc/5.1.0:gcc-10.1] { contexts: all dependencies: { gcc/10.1 } compiled with gcc (10.1) prefix: /home/user/sw/libxc/5.1.0-gcc-10.1 standard paths: { bin: /home/user/sw/libxc/5.1.0-gcc-10.1/bin lib: /home/user/sw/libxc/5.1.0-gcc-10.1/lib include: /home/user/sw/libxc/5.1.0-gcc-10.1/include pkgConfig: /home/user/sw/libxc/5.1.0-gcc-10.1/lib/pkgconfig } } [libxc/5.1.0:intel-2020] { contexts: all dependencies: { intel/2020 } compiled with Intel icc (2020) prefix: /home/user/sw/libxc/5.1.0-intel-2020 standard paths: { bin: /home/user/sw/libxc/5.1.0-intel-2020/bin lib: /home/user/sw/libxc/5.1.0-intel-2020/lib64 include: /home/user/sw/libxc/5.1.0-intel-2020/include } } } } The file had no errors in its YAML syntax. Notice also that the standard paths (''bin'', ''lib64'', ''include'') are found and noted by VALET! ==== Runtime environment ==== To load a specific variant of **libxc** 5.1.0 into the runtime environment, the ''vpkg_require'' command is used: [user@login00.darwin ~]$ vpkg_require libxc/5.1.0:intel-2020 Adding dependency `intel/2020u4` to your environment Adding package `libxc/5.1.0:intel-2020` to your environment [user@login00.darwin ~]$ which xc-info ~/sw/libxc/5.1.0-intel-2020/bin/xc-info The ''xc-info'' command is used //without a leading path// which implies that the shell with check directories in the ''$PATH'' environment variable for an executable with that name. If a different version/variant of **libxc** is chosen: [frey@login00.darwin ~]$ vpkg_rollback all [frey@login00.darwin ~]$ vpkg_require libxc/5.1.0 Adding package `libxc/5.1.0` to your environment [frey@login00.darwin ~]$ which xc-info ~/sw/libxc/5.1.0/bin/xc-info The command is still ''xc-info'' but the shell finds it at a different location. This abstraction (no full paths to executables) makes it easier to alter complex job scripts by simply changing which variant is added using ''vpkg_require''. Note that the ''$LD_LIBRARY_PATH'' is augmented, as well: [user@login00.darwin ~]$ echo $LD_LIBRARY_PATH /home/user/sw/libxc/5.1.0/lib:/opt/shared/slurm/lib versus [user@login00.darwin ~]$ vpkg_rollback all [user@login00.darwin ~]$ vpkg_require libxc/5.1.0:intel-2020 Adding dependency `intel/2020u4` to your environment Adding package `libxc/5.1.0:intel-2020` to your environment [user@login00.darwin ~]$ echo $LD_LIBRARY_PATH /home/user/sw/libxc/5.1.0-intel-2020/lib64:/opt/shared/intel/2020u4/compilers_and_libraries_2020.4.304[....] ==== Development context ==== All actions taken by VALET occur in a //context//. The //context// is an arbitrary string, like ''development'', that limits what actions are taken in the runtime environment. By default, an action happens regardless of the context. But there are several actions VALET produces by default that apply to the aforementioned ''development'' context: * An additional environment variable is set to the prefix directory for the version/variant * The ''/include'' directory is appended to the ''$CPPFLAGS'' environment variable (e.g. ''-I/include'') * The ''/lib'' and/or ''/lib64'' directories are appended to the ''$LDFLAGS'' environment variable (e.g. ''-L/lib64'') For the **libxc** package: [user@login00.darwin ~]$ vpkg_rollback all [user@login00.darwin ~]$ vpkg_require --context=development libxc/5.1.0 Adding package `libxc/5.1.0` to your environment [user@login00.darwin ~]$ echo $LIBXC_PREFIX /home/user/sw/libxc/5.1.0 [user@login00.darwin ~]$ echo $CPPFLAGS -I/home/user/sw/libxc/5.1.0/include [user@login00.darwin ~]$ echo $LDFLAGS -L/home/user/sw/libxc/5.1.0/lib As mentioned at the start of this document, the ''$LIBXC_PREFIX'' can be particularly helpful when configuring an Autoconf or CMake build of software that depends on this version/variant of **libxc**. The development context is the most common use of context in VALET that it can be shortened from ''vpkg_require --context=development'' to just ''vpkg_devrequire''. ===== Recipes for Specific Software ===== All recipes are provided based on a specific cluster, Caviness and/or DARWIN. However each recipe may be used as a reference from one cluster to another by using the appropriate file system (directory structure) and VALET packages for the cluster you are trying to install the software. ==== MCFOST ==== [[technical:recipes:mcfost|Building MCFOST on Caviness]] ==== Python ==== * [[technical:recipes:pyqt5-in-virtualenv|Building PyQt5 in a Python Virtual Environment]] * [[technical:recipes:jupyter-notebook|Jupyter Notebook Python Virtual Environment]] * [[technical:recipes:keras-in-virtualenv|Keras Python Virtual Environment]] * Gurobi Python Virtual Environments * [[software:gurobi:caviness#serial-or-threaded-python-environment|Gurobi Python Threaded and Serial]] * [[software:gurobi:caviness#mpi-python-environment|Gurobi Python MPI]] * [[technical:recipes:emcee-in-virtualenv|Python Virtualenv: emcee and pyKLIP]] * [[technical:recipes:mpi4py-in-virtualenv|Python Virtual Environments with mpi4py]] * [[technical:recipes:tensorflow-in-virtualenv|TensorFlow Python Virtual Environment]] ==== WRF ==== * [[software:wrf:caviness|WRF on Caviness]] * [[software:wrf:darwin|WRF on DARWIN]] ==== VASP ==== [[technical:recipes:vasp-6-darwin|Building VASP 6 on Caviness/DARWIN]]