Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revisionBoth sides next revision | ||
technical:recipes:software-managment [2021-02-16 09:01] – [Preparations] anita | technical:recipes:software-managment [2023-03-09 14:01] – [Python] anita | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== Software Management ====== | ||
- | This document offers an introduction to the mechanisms — organizational and procedural — used by IT RCI staff to build and manage the software made available to users on the HPC systems: | ||
- | |||
- | === Organizational === | ||
- | |||
- | It is most often the case that you do not use a single version of a piece of software. | ||
- | |||
- | Organization is also important in maintaining // | ||
- | * Maintaining a full copy of your source code with each calculation: | ||
- | * Copying executables and libraries into each working directory: | ||
- | * Including extensive Unix environment manipulation in every job script: | ||
- | |||
- | === Procedural === | ||
- | |||
- | For many versions and variants of software to coexist on a system and be used properly, users cannot just modify their login files (e.g. '' | ||
- | |||
- | VALET also makes it easier to build software. | ||
- | <code bash> | ||
- | [user@login00.darwin ~]$ vpkg_devrequire udunits/ | ||
- | Adding package `udunits/ | ||
- | [user@login00.darwin ~]$ echo $UDUNITS_PREFIX | ||
- | / | ||
- | [user@login00.darwin ~]$ echo $LDFLAGS | ||
- | -L/ | ||
- | [user@login00.darwin ~]$ echo $CPPFLAGS | ||
- | -I/ | ||
- | </ | ||
- | The GNU Autoconf build system makes use of these environment variables when searching for and building software, and other tools can make use of the value of the variable (e.g. '' | ||
- | |||
- | VALET is not just a tool that system administrators can use to describe environment changes: | ||
- | |||
- | ===== Preparations ===== | ||
- | |||
- | In this example we will build the latest version of the [[https:// | ||
- | |||
- | First and foremost, decide where on the file system you are going to organize the set of versions/ | ||
- | <code bash> | ||
- | [user@login00.darwin ~]$ LIBXC_BASEDIR=~/ | ||
- | </ | ||
- | For shared installations made available to your entire workgroup: | ||
- | <code bash> | ||
- | [(workgroup: | ||
- | </ | ||
- | |||
- | Ensure the base directory exists: | ||
- | <code bash> | ||
- | [user@login00.darwin ~]$ mkdir -p " | ||
- | </ | ||
- | |||
- | Often the software being built is distributed as one or more downloadable files (e.g. tar.gz archive). | ||
- | |||
- | <code bash> | ||
- | [user@login00.darwin ~]$ mkdir -p " | ||
- | [user@login00.darwin ~]$ wget -O " | ||
- | " | ||
- | : | ||
- | </ | ||
- | |||
- | At this point the source code has been downloaded, our directory hierarchy has been established, | ||
- | |||
- | ===== Builds ===== | ||
- | |||
- | Each of the compilers chosen in building // | ||
- | |||
- | ==== System GCC ==== | ||
- | |||
- | The cluster comes with a native GNU toolchain present (gcc, g++, gfortran). | ||
- | |||
- | Prepare the variant' | ||
- | <code bash> | ||
- | [user@login00.darwin ~]$ mkdir " | ||
- | [user@login00.darwin ~]$ cd " | ||
- | [user@login00.darwin 5.1.0]$ tar -xf " | ||
- | [user@login00.darwin 5.1.0]$ mv libxc-5.1.0 src | ||
- | [user@login00.darwin 5.1.0]$ cd src | ||
- | </ | ||
- | |||
- | The **libxc** build system uses GNU Autoconf or CMake. | ||
- | < | ||
- | [user@login00.darwin src]$ ./configure --prefix=" | ||
- | </ | ||
- | Additional flags may be necessary, but the most important is the '' | ||
- | <code bash> | ||
- | [user@login00.darwin src]$ make | ||
- | : | ||
- | xc-threshold.c: | ||
- | xc-threshold.c: | ||
- | for (int i = 0; i < (int) (sizeof(xc_values_type) / sizeof(double)); | ||
- | ^ | ||
- | xc-threshold.c: | ||
- | </ | ||
- | This build encountered an error: | ||
- | <code bash> | ||
- | [user@login00.darwin src]$ CFLAGS=" | ||
- | : | ||
- | [user@login00.darwin src]$ make | ||
- | : | ||
- | [user@login00.darwin src]$ make install | ||
- | : | ||
- | </ | ||
- | |||
- | ==== Intel ==== | ||
- | |||
- | The Intel compilers offer advanced optimizations to target the specific processor models' | ||
- | <code bash> | ||
- | [user@login00.darwin ~]$ cd | ||
- | [user@login00.darwin ~]$ vpkg_require intel/ | ||
- | Adding package `intel/ | ||
- | [user@login00.darwin ~]$ which icc | ||
- | / | ||
- | </ | ||
- | We will also use CMake for this build; though the OS does provide a version of CMake, it is relatively old so a newer version is often necessary: | ||
- | <code bash> | ||
- | [user@login00.darwin ~]$ vpkg_require cmake/ | ||
- | Adding package `cmake/ | ||
- | </ | ||
- | |||
- | Since this variant of 5.1.0 uses the Intel compilers, it will have a version identifier that includes a //feature// named '' | ||
- | <code bash> | ||
- | [user@login00.darwin ~]$ vpkg_id2path --version=" | ||
- | 5.1.0-intel-2020 | ||
- | </ | ||
- | Leading to the unpack procedure: | ||
- | <code bash> | ||
- | [user@login00.darwin ~]$ mkdir " | ||
- | [user@login00.darwin ~]$ cd " | ||
- | [user@login00.darwin 5.1.0-intel-2020]$ tar -xf " | ||
- | [user@login00.darwin 5.1.0-intel-2020]$ mv libxc-5.1.0 src | ||
- | [user@login00.darwin 5.1.0-intel-2020]$ cd src | ||
- | </ | ||
- | |||
- | CMake builds usually request that you create an empty directory to hold all of the files generated by CMake. | ||
- | <code bash> | ||
- | [user@login00.darwin 5.1.0-intel-2020]$ mkdir build | ||
- | [user@login00.darwin 5.1.0-intel-2020]$ cd build | ||
- | [user@login00.darwin build]$ CC=icc FC=ifort CXX=icpc cmake \ | ||
- | | ||
- | | ||
- | .. | ||
- | : | ||
- | </ | ||
- | The CMake build system doesn' | ||
- | <code bash> | ||
- | [user@login00.darwin build]$ make | ||
- | : | ||
- | [user@login00.darwin build]$ make install | ||
- | : | ||
- | </ | ||
- | |||
- | ==== GCC 10 ==== | ||
- | |||
- | Finally, a variant using the GCC 10 compiler will be built using Autoconf. | ||
- | <code bash> | ||
- | [user@login00.darwin build]$ cd | ||
- | [user@login00.darwin ~]$ vpkg_rollback all | ||
- | </ | ||
- | and configure the environment for GCC 10 and the newer CMake: | ||
- | <code bash> | ||
- | [user@login00.darwin ~]$ vpkg_versions gcc | ||
- | |||
- | Available versions in package (* = default version): | ||
- | |||
- | [/ | ||
- | gcc GCC Compiler Suite | ||
- | 4.8 alias to gcc/4.8.5 | ||
- | 4.8.5 | ||
- | 7.3 alias to gcc/7.3.0 | ||
- | 7.3.0 GCC with C, C++, Obj-C, Obj-C++, Fortran, and GO | ||
- | 10.1.0 | ||
- | 10.1 alias to gcc/10.1.0 | ||
- | * system | ||
- | [user@login00.darwin ~]$ vpkg_require cmake/ | ||
- | Adding package `cmake/ | ||
- | Adding package `gcc/ | ||
- | </ | ||
- | <WRAP center round tip 60%> | ||
- | Notice that multiple packages can be specified in a single '' | ||
- | </ | ||
- | As with the Intel example, we will add a '' | ||
- | <code bash> | ||
- | [user@login00.darwin ~]$ vpkg_id2path --version=" | ||
- | 5.1.0-gcc-10.1 | ||
- | [user@login00.darwin ~]$ mkdir " | ||
- | [user@login00.darwin ~]$ cd " | ||
- | [user@login00.darwin 5.1.0-gcc-10.1]$ tar -xf " | ||
- | [user@login00.darwin 5.1.0-gcc-10.1]$ mv libxc-5.1.0 src | ||
- | [user@login00.darwin 5.1.0-gcc-10.1]$ cd src | ||
- | </ | ||
- | Sometimes Autoconf allows the same approach as CMake: | ||
- | <code bash> | ||
- | [user@login00.darwin src]$ mkdir build | ||
- | [user@login00.darwin src]$ cd build | ||
- | [user@login00.darwin build]$ CC=gcc FC=gfortran ../ | ||
- | : | ||
- | [user@login00.darwin build]$ make | ||
- | : | ||
- | [user@login00.darwin build]$ make install | ||
- | : | ||
- | </ | ||
- | <WRAP center round info 60%> | ||
- | Note that with GCC 10 the additional '' | ||
- | </ | ||
- | |||
- | ==== Results ==== | ||
- | |||
- | Through the course of the above sections three distinct variants of **libxc** 5.1.0 were produced. | ||
- | <code bash> | ||
- | [user@login00.darwin build]$ cd | ||
- | [user@login00.darwin ~]$ vpkg_rollback all | ||
- | [user@login00.darwin ~]$ ls -l " | ||
- | total 21 | ||
- | drwxr-xr-x 6 user everyone 6 Feb 8 11:35 5.1.0 | ||
- | drwxr-xr-x 6 user everyone 6 Feb 8 12:11 5.1.0-gcc-10.1 | ||
- | drwxr-xr-x 7 user everyone 7 Feb 8 11:56 5.1.0-intel-2020 | ||
- | drwxr-xr-x 2 user everyone 3 Feb 8 11:10 attic | ||
- | </ | ||
- | Each variant is structured similarly, with the typical installation directory layout: | ||
- | <code bash> | ||
- | [user@login00.darwin ~]$ ls -l " | ||
- | total 40 | ||
- | drwxr-xr-x | ||
- | drwxr-xr-x | ||
- | drwxr-xr-x | ||
- | drwxr-xr-x | ||
- | drwxr-xr-x 11 user everyone 40 Feb 8 11:50 src | ||
- | </ | ||
- | |||
- | To make use of one of these variants of **libxc**, these directories must be added to specific environment variables: a task for which VALET is designed to assist. | ||
- | |||
- | ===== VALET Setup ===== | ||
- | |||
- | Since **libxc** adheres to the standard directory layout for software on Linux ('' | ||
- | * '' | ||
- | * '' | ||
- | In addition, for software-building the linker could be told to check '' | ||
- | |||
- | VALET automatically recognizes the standard directory layout, so configuring these variants of '' | ||
- | <code bash> | ||
- | [user@login00.darwin ~]$ echo $LIBXC_BASEDIR | ||
- | / | ||
- | </ | ||
- | Since these builds were done in the user's home directory, they were personal copies of the software and should use a //VALET package definition file// stored in '' | ||
- | <code bash> | ||
- | [user@login00.darwin ~]$ VALET_PKG_DIR=~/ | ||
- | </ | ||
- | versus an installation made for an entire workgroup, which would store package definition files in '' | ||
- | <code bash> | ||
- | [user@login00.darwin ~]$ VALET_PKG_DIR=" | ||
- | </ | ||
- | Whichever scheme is in-use, ensure the directory exists: | ||
- | <code bash> | ||
- | [user@login00.darwin ~]$ mkdir -p --mode=$VALET_PKG_DIR_MODE " | ||
- | </ | ||
- | |||
- | VALET allows package definitions in a variety of formats (XML, JSON, YAML) but YAML tends to be the simplest format so we will use it here. | ||
- | |||
- | ==== Package section ==== | ||
- | |||
- | The //package section// of the definition file includes items that apply to all versions/ | ||
- | <code yaml> | ||
- | libxc: | ||
- | prefix: / | ||
- | description: | ||
- | url: " | ||
- | </ | ||
- | The //package identifier// | ||
- | |||
- | ==== Versions ==== | ||
- | |||
- | The '' | ||
- | <code yaml> | ||
- | libxc: | ||
- | prefix: / | ||
- | description: | ||
- | url: " | ||
- | | ||
- | versions: | ||
- | " | ||
- | description: | ||
- | </ | ||
- | |||
- | <WRAP center round tip 60%> | ||
- | Since we used '' | ||
- | |||
- | The implicit behavior is overridden by providing a '' | ||
- | </ | ||
- | |||
- | For the other two variants, a dependency exists with respect to the compiler toolchain that was used. Each of those variants should be defined with that dependency described: | ||
- | <code yaml> | ||
- | libxc: | ||
- | prefix: / | ||
- | description: | ||
- | url: " | ||
- | | ||
- | versions: | ||
- | " | ||
- | description: | ||
- | " | ||
- | description: | ||
- | dependencies: | ||
- | - intel/2020 | ||
- | " | ||
- | description: | ||
- | dependencies: | ||
- | - gcc/10.1 | ||
- | </ | ||
- | |||
- | <WRAP center round tip 60%> | ||
- | We could have been more specific about the version of the Intel and GCC compiler that was used, e.g. '' | ||
- | </ | ||
- | |||
- | Finally, it is a good idea to specify which version definition should act as the default. | ||
- | <file libxc.vpkg_yaml> | ||
- | libxc: | ||
- | prefix: / | ||
- | description: | ||
- | url: " | ||
- | default-version: | ||
- | versions: | ||
- | " | ||
- | description: | ||
- | " | ||
- | description: | ||
- | dependencies: | ||
- | - intel/2020 | ||
- | " | ||
- | description: | ||
- | dependencies: | ||
- | - gcc/10.1 | ||
- | </ | ||
- | saved at '' | ||
- | |||
- | ==== Checking the definition file ==== | ||
- | |||
- | The package definition file can be syntax-checked: | ||
- | <code bash> | ||
- | [user@login00.darwin ~]$ vpkg_check " | ||
- | / | ||
- | |||
- | [libxc] { | ||
- | contexts: all | ||
- | actions: { | ||
- | LIBXC_PREFIX=${VALET_PATH_PREFIX} (contexts: development) | ||
- | } | ||
- | https:// | ||
- | a library of exchange-correlation functionals for density-functional theory | ||
- | prefix: / | ||
- | source file: / | ||
- | default version: libxc/ | ||
- | versions: { | ||
- | [libxc/ | ||
- | contexts: all | ||
- | compiled with system gcc/ | ||
- | prefix: / | ||
- | standard paths: { | ||
- | bin: / | ||
- | lib: / | ||
- | include: / | ||
- | pkgConfig: / | ||
- | } | ||
- | } | ||
- | [libxc/ | ||
- | contexts: all | ||
- | dependencies: | ||
- | gcc/10.1 | ||
- | } | ||
- | compiled with gcc (10.1) | ||
- | prefix: / | ||
- | standard paths: { | ||
- | bin: / | ||
- | lib: / | ||
- | include: / | ||
- | pkgConfig: / | ||
- | } | ||
- | } | ||
- | [libxc/ | ||
- | contexts: all | ||
- | dependencies: | ||
- | intel/2020 | ||
- | } | ||
- | compiled with Intel icc (2020) | ||
- | prefix: / | ||
- | standard paths: { | ||
- | bin: / | ||
- | lib: / | ||
- | include: / | ||
- | } | ||
- | } | ||
- | } | ||
- | } | ||
- | </ | ||
- | The file had no errors in its YAML syntax. | ||
- | |||
- | ==== Runtime environment ==== | ||
- | |||
- | To load a specific variant of **libxc** 5.1.0 into the runtime environment, | ||
- | <code bash> | ||
- | [user@login00.darwin ~]$ vpkg_require libxc/ | ||
- | Adding dependency `intel/ | ||
- | Adding package `libxc/ | ||
- | [user@login00.darwin ~]$ which xc-info | ||
- | ~/ | ||
- | </ | ||
- | The '' | ||
- | <code bash> | ||
- | [frey@login00.darwin ~]$ vpkg_rollback all | ||
- | [frey@login00.darwin ~]$ vpkg_require libxc/5.1.0 | ||
- | Adding package `libxc/ | ||
- | [frey@login00.darwin ~]$ which xc-info | ||
- | ~/ | ||
- | </ | ||
- | The command is still '' | ||
- | |||
- | Note that the '' | ||
- | <code bash> | ||
- | [user@login00.darwin ~]$ echo $LD_LIBRARY_PATH | ||
- | / | ||
- | </ | ||
- | versus | ||
- | <code bash> | ||
- | [user@login00.darwin ~]$ vpkg_rollback all | ||
- | [user@login00.darwin ~]$ vpkg_require libxc/ | ||
- | Adding dependency `intel/ | ||
- | Adding package `libxc/ | ||
- | [user@login00.darwin ~]$ echo $LD_LIBRARY_PATH | ||
- | / | ||
- | </ | ||
- | |||
- | ==== Development context ==== | ||
- | |||
- | All actions taken by VALET occur in a // | ||
- | * An additional environment variable is set to the prefix directory for the version/ | ||
- | * The ''< | ||
- | * The ''< | ||
- | For the **libxc** package: | ||
- | <code bash> | ||
- | [user@login00.darwin ~]$ vpkg_rollback all | ||
- | [user@login00.darwin ~]$ vpkg_require --context=development libxc/5.1.0 | ||
- | Adding package `libxc/ | ||
- | [user@login00.darwin ~]$ echo $LIBXC_PREFIX | ||
- | / | ||
- | [user@login00.darwin ~]$ echo $CPPFLAGS | ||
- | -I/ | ||
- | [user@login00.darwin ~]$ echo $LDFLAGS | ||
- | -L/ | ||
- | </ | ||
- | As mentioned at the start of this document, the '' |