Software Management

This document offers an introduction to the mechanisms — organizational and procedural — used by IT RCI staff to build and manage the software made available to users on the HPC systems:

Organizational

It is most often the case that you do not use a single version of a piece of software. Bug fixes and feature additions drive change in software over time, leading to versioned releases of the product. In many cases the same versioned release may have mutually exclusive features that require multiple variants of that version to be maintained (simplest example: differing compiler toolchains, GCC versus Intel).

Organization is also important in maintaining reproducibility of your work. An unstable, difficult to maintain computing platform is the product of:

Maintaining a full copy of your source code with each calculation: minor to massive amounts of storage space can be wasted, and keeping multiple source trees in-sync is a challenge
Copying executables and libraries into each working directory: difficult to remain consistent on release version, feature set, build parameters, etc. involved in producing the executable
Including extensive Unix environment manipulation in every job script: any correction to the environment manipulation must be applied to every affected job script to remain consistent

Procedural

For many versions and variants of software to coexist on a system and be used properly, users cannot just modify their login files (e.g. .bashrc) to alter environment variables: such change affects every shell the user subsequently launches. Environment management tools like VALET encapsulate the changes so they can be made consistently and easily. Corrections can be made to the encapsulated change information to affect globally (for that version or variant of the software).

VALET also makes it easier to build software. Standard development paths — like the <prefix>/lib and <prefix>/include directories — are automatically added to key environment variables when the development context is used:

[user@login00.darwin ~]$ vpkg_devrequire udunits/2.2.28
Adding package `udunits/2.2.28` to your environment
[user@login00.darwin ~]$ echo $UDUNITS_PREFIX
/opt/shared/udunits/2.2.28
[user@login00.darwin ~]$ echo $LDFLAGS
-L/opt/shared/udunits/2.2.28/lib
[user@login00.darwin ~]$ echo $CPPFLAGS
-I/opt/shared/udunits/2.2.28/include

The GNU Autoconf build system makes use of these environment variables when searching for and building software, and other tools can make use of the value of the variable (e.g. cmake -DUDUNITS_ROOT=$UDUNITS_PREFIX).

VALET is not just a tool that system administrators can use to describe environment changes: all users can create their own package definitions for themselves (in their ~/.valet directory) or for their workgroup (in the $WORKDIR/sw/valet directory). A package definition can reference other packages as dependencies and have those dependencies automatically loaded into the environment.

In this example we will build the latest version of the libxc library using various compilers.

First and foremost, decide where on the file system you are going to organize the set of versions/variants of the software. For a user maintaining personal builds of the software the home directory can be used:

[user@login00.darwin ~]$ LIBXC_BASEDIR=~/sw/libxc

For shared installations made available to your entire workgroup:

On DARWIN:

[(workgroup:user)@login00.darwin ~]$ LIBXC_BASEDIR="${WORKDIR_SW}/libxc"

On Caviness:

[(workgroup:user)@login00 ~]$ LIBXC_BASEDIR="${WORKDIR}/sw/libxc"

Ensure the base directory exists:

[user@login00.darwin ~]$ mkdir -p "$LIBXC_BASEDIR"

Often the software being built is distributed as one or more downloadable files (e.g. tar.gz archive). IT RCI staff create a directory to hold (organize!) these files. For libxc we will create that directory and download the source archive:

[user@login00.darwin ~]$ mkdir -p "$LIBXC_BASEDIR/attic"
[user@login00.darwin ~]$ wget -O "$LIBXC_BASEDIR/attic/libxc-5.1.0.tar-gz" \
       "http://www.tddft.org/programs/libxc/down.php?file=5.1.0/libxc-5.1.0.tar.gz"

At this point the source code has been downloaded, our directory hierarchy has been established, and we are ready to build a variant of the 5.1.0 release.

Each of the compilers chosen in building variants of the 5.1.0 release will follow the same general procedure.

The notation : (colon) in output examples throughout this document indicates lines of output are displayed by the command entered but have been omitted from this documentation.

The cluster comes with a native GNU toolchain present (gcc, g++, gfortran). No environment changes are necessary to use these compilers. Since it is the default compiler toolchain, IT RCI staff do not mention it when naming a software variant: this variant will have a version identifier of 5.1.0.

Prepare the variant's installation directory and unpack the source code inside it:

[user@login00.darwin ~]$ mkdir "$LIBXC_BASEDIR/5.1.0"
[user@login00.darwin ~]$ cd "$LIBXC_BASEDIR/5.1.0"
[user@login00.darwin 5.1.0]$ tar -xf "$LIBXC_BASEDIR/attic/libxc-5.1.0.tar-gz"
[user@login00.darwin 5.1.0]$ mv libxc-5.1.0 src
[user@login00.darwin 5.1.0]$ cd src

The libxc build system uses GNU Autoconf or CMake. For this variant, we will use Autoconf:

[user@login00.darwin src]$ ./configure --prefix="$LIBXC_BASEDIR/5.1.0"

Additional flags may be necessary, but the most important is the –prefix, telling the build system the software is intended to be installed into the chosen directory for the variant.

[user@login00.darwin src]$ make
   :
xc-threshold.c: In function ‘check_xc’:
xc-threshold.c:808:3: error: ‘for’ loop initial declarations are only allowed in C99 mode
   for (int i = 0; i < (int) (sizeof(xc_values_type) / sizeof(double)); i++)
   ^
xc-threshold.c:808:3: note: use option -std=c99 or -std=gnu99 to compile your code

This build encountered an error: the system GNU C compiler defaults to an older language standard than what the source code implies, so the compiler has mentioned that a flag is necessary to choose that standard:

[user@login00.darwin src]$ CFLAGS="-std=gnu99" ./configure --prefix="$LIBXC_BASEDIR/5.1.0"
   :
[user@login00.darwin src]$ make
   :
[user@login00.darwin src]$ make install
   :

The Intel compilers offer advanced optimizations to target the specific processor models' features. To use the Intel compilers they must be first added to the environment:

[user@login00.darwin ~]$ cd
[user@login00.darwin ~]$ vpkg_require intel/2020u4
Adding package `intel/2020u4` to your environment
[user@login00.darwin ~]$ which icc
/opt/shared/intel/2020u4/compilers_and_libraries_2020.4.304/linux/bin/intel64/icc

We will also use CMake for this build; though the OS does provide a version of CMake, it is relatively old so a newer version is often necessary:

[user@login00.darwin ~]$ vpkg_require cmake/3.19.1
Adding package `cmake/3.19.1` to your environment

Since this variant of 5.1.0 uses the Intel compilers, it will have a version identifier that includes a feature named intel-2020. The appropriate directory name for that identifier can be determined using:

[user@login00.darwin ~]$ vpkg_id2path --version="5.1.0:intel-2020"
5.1.0-intel-2020

Leading to the unpack procedure:

[user@login00.darwin ~]$ mkdir "$LIBXC_BASEDIR/5.1.0-intel-2020"
[user@login00.darwin ~]$ cd "$LIBXC_BASEDIR/5.1.0-intel-2020"
[user@login00.darwin 5.1.0-intel-2020]$ tar -xf "$LIBXC_BASEDIR/attic/libxc-5.1.0.tar-gz"
[user@login00.darwin 5.1.0-intel-2020]$ mv libxc-5.1.0 src
[user@login00.darwin 5.1.0-intel-2020]$ cd src

CMake builds usually request that you create an empty directory to hold all of the files generated by CMake. The commands analogous to the Autoconf setup of the build look like:

[user@login00.darwin 5.1.0-intel-2020]$ mkdir build
[user@login00.darwin 5.1.0-intel-2020]$ cd build
[user@login00.darwin build]$ CC=icc FC=ifort CXX=icpc cmake \
       -DCMAKE_INSTALL_PREFIX="$LIBXC_BASEDIR/5.1.0-intel-2020" \
       -DBUILD_TESTING=FALSE \
       ..
   :

The CMake build system doesn't yet support Fortran compilation 100%, so by default that language is omitted (and the FC=ifort is probably extraneous). By the same token, none of the source is written in C++ so specifying CXX=icpc is also extraneous, but in neither case does it hurt to have your intentions clear.

[user@login00.darwin build]$ make
   :
[user@login00.darwin build]$ make install
   :

Finally, a variant using the GCC 10 compiler will be built using Autoconf. First, remove all environment changes made for the Intel build

[user@login00.darwin build]$ cd
[user@login00.darwin ~]$ vpkg_rollback all

and configure the environment for GCC 10 and the newer CMake:

[user@login00.darwin ~]$ vpkg_versions gcc
 
Available versions in package (* = default version):
 
[/opt/shared/valet/2.1/etc/gcc.vpkg_yaml]
gcc       GCC Compiler Suite
  4.8     alias to gcc/4.8.5
  4.8.5   CentOS system GCC with C, C++, Obj-C, Obj-C++, and Fortran
  7.3     alias to gcc/7.3.0
  7.3.0   GCC with C, C++, Obj-C, Obj-C++, Fortran, and GO
  10.1.0  GCC with C, C++, Obj-C, Obj-C++, Fortran, and GO
  10.1    alias to gcc/10.1.0
* system  alias to gcc/4.8.5
[user@login00.darwin ~]$ vpkg_require cmake/3.19.1 gcc/10.1.0
Adding package `cmake/3.19.1` to your environment
Adding package `gcc/10.1.0` to your environment

Notice that multiple packages can be specified in a single vpkg_require command — this is more efficient than running them as multiple vpkg_require commands.

As with the Intel example, we will add a gcc-10.1 feature to the version id to indicate the compiler choice, so the preparations look like:

[user@login00.darwin ~]$ vpkg_id2path --version="5.1.0:gcc-10.1"
5.1.0-gcc-10.1
[user@login00.darwin ~]$ mkdir "$LIBXC_BASEDIR/5.1.0-gcc-10.1"
[user@login00.darwin ~]$ cd "$LIBXC_BASEDIR/5.1.0-gcc-10.1"
[user@login00.darwin 5.1.0-gcc-10.1]$ tar -xf "$LIBXC_BASEDIR/attic/libxc-5.1.0.tar-gz"
[user@login00.darwin 5.1.0-gcc-10.1]$ mv libxc-5.1.0 src
[user@login00.darwin 5.1.0-gcc-10.1]$ cd src

Sometimes Autoconf allows the same approach as CMake: creating a build directory to hold the files produced by the build system, leaving the source directory undisturbed.

[user@login00.darwin src]$ mkdir build
[user@login00.darwin src]$ cd build
[user@login00.darwin build]$ CC=gcc FC=gfortran ../configure --prefix="$LIBXC_BASEDIR/5.1.0-gcc-10.1"
   :
[user@login00.darwin build]$ make
   :
[user@login00.darwin build]$ make install
   :

Note that with GCC 10 the additional CFLAGS=-std=gnu99 was not present on the ../configure command. With the 10.1 gcc compiler, the default language standard is listed as gnu18. The default for GCC 4.8.5 is gnu89 — older than the C99 standard the code requires.

Through the course of the above sections three distinct variants of libxc 5.1.0 were produced. The base directory currently looks like:

[user@login00.darwin build]$ cd
[user@login00.darwin ~]$ vpkg_rollback all
[user@login00.darwin ~]$ ls -l "$LIBXC_BASEDIR"
total 21
drwxr-xr-x 6 user everyone 6 Feb  8 11:35 5.1.0
drwxr-xr-x 6 user everyone 6 Feb  8 12:11 5.1.0-gcc-10.1
drwxr-xr-x 7 user everyone 7 Feb  8 11:56 5.1.0-intel-2020
drwxr-xr-x 2 user everyone 3 Feb  8 11:10 attic

Each variant is structured similarly, with the typical installation directory layout:

[user@login00.darwin ~]$ ls -l "$LIBXC_BASEDIR/5.1.0-intel-2020"
total 40
drwxr-xr-x  2 user everyone  3 Feb  8 11:56 bin
drwxr-xr-x  2 user everyone  7 Feb  8 11:56 include
drwxr-xr-x  3 user everyone  4 Feb  8 11:56 lib64
drwxr-xr-x  3 user everyone  3 Feb  8 11:56 share
drwxr-xr-x 11 user everyone 40 Feb  8 11:50 src

To make use of one of these variants of libxc, these directories must be added to specific environment variables: a task for which VALET is designed to assist.

Since libxc adheres to the standard directory layout for software on Linux (bin, lib or lib64, include) it is relatively easy to setup the runtime environment:

$LIBXC_BASEDIR/5.1.0-intel-2020/bin must be added to $PATH
$LIBXC_BASEDIR/5.1.0-intel-2020/lib64 must be added to $LD_LIBRARY_PATH

In addition, for software-building the linker could be told to check $LIBXC_BASEDIR/5.1.0-intel-2020/lib64 for required libraries and the C/C++ compiler told to look in $LIBXC_BASEDIR/5.1.0-intel-2020/include for header files — more on that in a moment.

VALET automatically recognizes the standard directory layout, so configuring these variants of libxc is very straightforward. First, note where the variants were collocated on the file system:

[user@login00.darwin ~]$ echo $LIBXC_BASEDIR 
/home/user/sw/libxc

Since these builds were done in the user's home directory, they were personal copies of the software and should use a VALET package definition file stored in ~/.valet

[user@login00.darwin ~]$ VALET_PKG_DIR=~/.valet ; VALET_PKG_DIR_MODE=0700

versus an installation made for an entire workgroup would store VALET package definition files in

[user@login00.darwin ~]$ VALET_PKG_DIR="$WORKDIR_SW/valet"

on DARWIN and in

[user@login00.darwin ~]$ VALET_PKG_DIR="$WORKDIR/sw/valet" ; VALET_PKG_DIR_MODE=2770

on Caviness.

Whichever scheme is in-use, ensure the directory exists for personal use on Caviness and DARWIN, and entire workgroup on Caviness (entire workgroup on DARWIN is automatically created for each allocation so this is not necessary)

[user@login00.darwin ~]$ mkdir -p --mode=$VALET_PKG_DIR_MODE "$VALET_PKG_DIR"

VALET allows package definitions in a variety of formats (XML, JSON, YAML) but YAML tends to be the simplest format so we will use it here.

The package section of the definition file includes items that apply to all versions/variants of the software:

libxc:
    prefix: /home/user/sw/libxc
    description: a library of exchange-correlation functionals for density-functional theory
    url: "https://tddft.org/programs/libxc/"

The package identifier is the top-level key in the document — libxc — and the value of $LIBXC_BASEDIR is the value of the prefix key in this section. The URL and description are information taken from the official libxc web site.

The versions key is used to provide a list of the versions/variants of the software. The simplest version that was built used the system GCC compiler:

libxc:
    prefix: /home/user/sw/libxc
    description: a library of exchange-correlation functionals for density-functional theory
    url: "https://tddft.org/programs/libxc/"
    
    versions:
        "5.1.0":
            description: compiled with system gcc/gfortran (4.8.5)

Since we used vpkg_id2path to determine the path associated with the version identifiers and each variant is installed under $LIBXC_BASEDIR, there's no need to specify a prefix for the version definitions: package's prefix (/home/user/sw/libxc) with the version identifier appended (/home/user/sw/libxc/5.1.0) is implicit.

The implicit behavior is overridden by providing a prefix key in the version definition: a relative path is appended to the package's prefix, an absolute path is used as-is.

For the other two variants, a dependency exists with respect to the compiler toolchain that was used. Each of those variants should be defined with that dependency described:

libxc:
    prefix: /home/user/sw/libxc
    description: a library of exchange-correlation functionals for density-functional theory
    url: "https://tddft.org/programs/libxc/"
    
    versions:
        "5.1.0":
            description: compiled with system gcc/gfortran (4.8.5)
        "5.1.0:intel-2020":
            description: compiled with Intel icc (2020)
            dependencies:
                - intel/2020
        "5.1.0:gcc-10.1":
            description: compiled with gcc (10.1)
            dependencies:
                - gcc/10.1

We could have been more specific about the version of the Intel and GCC compiler that was used, e.g. intel/2020u4 and gcc/10.1.0. Minor releases of software do not usually significantly change its behavior or alter APIs, so by being less specific (intel/2020) if the Intel compiler were upgraded to 2020u5 (and the version identifier 2020 promoted to point to that by the system administrator) then our libxc/5.1.0:intel-2020 would automatically have the new version of Intel as its dependency. Note that this automatic "upgrade" does not work for static executables and libraries (they would need to be rebuilt to use the new runtime libraries, for example).

Finally, it is a good idea to specify which version definition should act as the default. This yields the following package definition file

libxc:
    prefix: /home/user/sw/libxc
    description: a library of exchange-correlation functionals for density-functional theory
    url: "https://tddft.org/programs/libxc/"
    default-version: "5.1.0:gcc-10.1"
    versions:
        "5.1.0":
            description: compiled with system gcc/gfortran (4.8.5)
        "5.1.0:intel-2020":
            description: compiled with Intel icc (2020)
            dependencies:
                - intel/2020
        "5.1.0:gcc-10.1":
            description: compiled with gcc (10.1)
            dependencies:
                - gcc/10.1

saved at $VALET_PKG_DIR/libxc.vpkg_yaml.

The package definition file can be syntax-checked:

[user@login00.darwin ~]$ vpkg_check "$VALET_PKG_DIR/libxc.vpkg_yaml"
/home/user/.valet/libxc.vpkg_yaml is OK
 
[libxc] {
  contexts: all
  actions: {
    LIBXC_PREFIX=${VALET_PATH_PREFIX} (contexts: development)
  }
  https://tddft.org/programs/libxc/
  a library of exchange-correlation functionals for density-functional theory
  prefix: /home/user/sw/libxc
  source file: /home/user/.valet/libxc.vpkg_yaml
  default version: libxc/5.1.0:intel-2020
  versions: {
    [libxc/5.1.0] {
      contexts: all
      compiled with system gcc/gfortran (4.8.5)
      prefix: /home/user/sw/libxc/5.1.0
      standard paths: {
        bin: /home/user/sw/libxc/5.1.0/bin
        lib: /home/user/sw/libxc/5.1.0/lib
        include: /home/user/sw/libxc/5.1.0/include
        pkgConfig: /home/user/sw/libxc/5.1.0/lib/pkgconfig
      }
    }
    [libxc/5.1.0:gcc-10.1] {
      contexts: all
      dependencies: {
        gcc/10.1
      }
      compiled with gcc (10.1)
      prefix: /home/user/sw/libxc/5.1.0-gcc-10.1
      standard paths: {
        bin: /home/user/sw/libxc/5.1.0-gcc-10.1/bin
        lib: /home/user/sw/libxc/5.1.0-gcc-10.1/lib
        include: /home/user/sw/libxc/5.1.0-gcc-10.1/include
        pkgConfig: /home/user/sw/libxc/5.1.0-gcc-10.1/lib/pkgconfig
      }
    }
    [libxc/5.1.0:intel-2020] {
      contexts: all
      dependencies: {
        intel/2020
      }
      compiled with Intel icc (2020)
      prefix: /home/user/sw/libxc/5.1.0-intel-2020
      standard paths: {
        bin: /home/user/sw/libxc/5.1.0-intel-2020/bin
        lib: /home/user/sw/libxc/5.1.0-intel-2020/lib64
        include: /home/user/sw/libxc/5.1.0-intel-2020/include
      }
    }
  }
}

The file had no errors in its YAML syntax. Notice also that the standard paths (bin, lib64, include) are found and noted by VALET!

To load a specific variant of libxc 5.1.0 into the runtime environment, the vpkg_require command is used:

[user@login00.darwin ~]$ vpkg_require libxc/5.1.0:intel-2020
Adding dependency `intel/2020u4` to your environment
Adding package `libxc/5.1.0:intel-2020` to your environment
[user@login00.darwin ~]$ which xc-info
~/sw/libxc/5.1.0-intel-2020/bin/xc-info

The xc-info command is used without a leading path which implies that the shell with check directories in the $PATH environment variable for an executable with that name. If a different version/variant of libxc is chosen:

[frey@login00.darwin ~]$ vpkg_rollback all
[frey@login00.darwin ~]$ vpkg_require libxc/5.1.0
Adding package `libxc/5.1.0` to your environment
[frey@login00.darwin ~]$ which xc-info
~/sw/libxc/5.1.0/bin/xc-info

The command is still xc-info but the shell finds it at a different location. This abstraction (no full paths to executables) makes it easier to alter complex job scripts by simply changing which variant is added using vpkg_require.

Note that the $LD_LIBRARY_PATH is augmented, as well:

[user@login00.darwin ~]$ echo $LD_LIBRARY_PATH
/home/user/sw/libxc/5.1.0/lib:/opt/shared/slurm/lib

versus

[user@login00.darwin ~]$ vpkg_rollback all
[user@login00.darwin ~]$ vpkg_require libxc/5.1.0:intel-2020
Adding dependency `intel/2020u4` to your environment
Adding package `libxc/5.1.0:intel-2020` to your environment
[user@login00.darwin ~]$ echo $LD_LIBRARY_PATH
/home/user/sw/libxc/5.1.0-intel-2020/lib64:/opt/shared/intel/2020u4/compilers_and_libraries_2020.4.304[....]

Development context

All actions taken by VALET occur in a context. The context is an arbitrary string, like development, that limits what actions are taken in the runtime environment. By default, an action happens regardless of the context. But there are several actions VALET produces by default that apply to the aforementioned development context:

An additional environment variable is set to the prefix directory for the version/variant
The <prefix>/include directory is appended to the $CPPFLAGS environment variable (e.g. -I<prefix>/include)
The <prefix>/lib and/or <prefix>/lib64 directories are appended to the $LDFLAGS environment variable (e.g. -L<prefix>/lib64)

For the libxc package:

[user@login00.darwin ~]$ vpkg_rollback all
[user@login00.darwin ~]$ vpkg_require --context=development libxc/5.1.0
Adding package `libxc/5.1.0` to your environment
[user@login00.darwin ~]$ echo $LIBXC_PREFIX
/home/user/sw/libxc/5.1.0
[user@login00.darwin ~]$ echo $CPPFLAGS
-I/home/user/sw/libxc/5.1.0/include
[user@login00.darwin ~]$ echo $LDFLAGS
-L/home/user/sw/libxc/5.1.0/lib

As mentioned at the start of this document, the $LIBXC_PREFIX can be particularly helpful when configuring an Autoconf or CMake build of software that depends on this version/variant of libxc. The development context is the most common use of context in VALET that it can be shortened from vpkg_require –context=development to just vpkg_devrequire.

All recipes are provided based on a specific cluster, Caviness and/or DARWIN. However each recipe may be used as a reference from one cluster to another by using the appropriate file system (directory structure) and VALET packages for the cluster you are trying to install the software.

Building MCFOST on Caviness

Building PyQt5 in a Python Virtual Environment
Jupyter Notebook Python Virtual Environment
Keras Python Virtual Environment
Gurobi Python Virtual Environments
- Gurobi Python Threaded and Serial
- Gurobi Python MPI
Python Virtualenv: emcee and pyKLIP
Python Virtual Environments with mpi4py
TensorFlow Python Virtual Environment

Building VASP 6 on Caviness/DARWIN

Software Management

Organizational

Procedural

Preparations

Builds

System GCC

Intel

GCC 10

Results

VALET Setup

Package section

Versions

Checking the definition file

Runtime environment

Development context

Recipes for Specific Software

MCFOST

Python

WRF

VASP

hpc documentation