Managing multiple versions of revision-controlled repositories

Revision-controlled source code retains a historical record of the changes that a project has gone through. Milestones along this path typically represent distinct releases of the software (a.k.a. versions). Git has become an extremely popular revision-control system, partly because its embedding of all history in the local copy of the repository produces less round-trip access with the canonical repository: once a local repository has been cloned, every snapshot up to that date and time is readily available.

In the more typical release strategy, a separate source code archive (e.g. a tar.gz or tar.bz2 file) is present for each release containing only the source code associated with that milestone. Thus, at least one copy of each distinct version of that software package is unpacked for build purposes. There will be a great deal of repetition of information in all of those copies, which represents redundant storage that a Git repository (as a sequence of differences between snapshots) will not incur. So for a software package built from source contained in a Git repository, only one copy of the repository is ever necessary. As new snapshots become available, that single repository can be updated with a git pull against the canonical repository.

In this document, the management of a Git-based software package is outlined. Workgroup storage will be used, but a path in a user's home directory would also be permissible. Common Git language references are made such as commit, clone, tag, checkout etc. so if you are unfamiliar with Git, you may want to refer to Git Commands if you need more background information.

The software package, truchas-tpl, has tagged milestones (releases) present in its history: for example, v18. Any builds of such a release will have a version identifier equivalent to the tag minus the leading "v": in the example, 18.

Outside of the tagged revisions, a build could be performed on any commit in the history. One important commit that will likely be of interest is the HEAD of the history (the last commit made, the most recent update to the code). There are two possible versioning choices for the HEAD: the long or short commit hash, or the date on which the HEAD was used. In this example, the long commit hash will be used for the build but additional version aliases will be added to the package's VALET definition to associate the short commit hash and the date with the build.

To create the package's base directory and populate a local Git repository:

[(workgroup:user)@login01 ~]$ TRUCHAS_TPL_PREFIX="${WORKDIR}/sw/truchas-tpl"
[(workgroup:user)@login01 ~]$ mkdir -p --mode=2775 "$TRUCHAS_TPL_PREFIX"
[(workgroup:user)@login01 ~]$ cd "$TRUCHAS_TPL_PREFIX"
[(workgroup:user)@login01 truchas_tpl]$ git clone https://gitlab.com/truchas/truchas-tpl.git src
[(workgroup:user)@login01 truchas_tpl]$ cd src

The Git repository now resides in the path ${WORKDIR}/sw/truchas-tpl/src. Each version of the package that is built will be installed into a directory at the same level, e.g. ${WORKDIR}/sw/truchas-tpl/<version>.

First, examine the tagged commits available:

[(workgroup:user)@login01 src]$ git tag | sort --key=1.2 --numeric-sort
v1
v2
 :
v17
v18

A build of the v18 commit will be performed. Since this package uses CMake, start by creating an out-of-tree build directory:

[(workgroup:user)@login01 src]$ mkdir build-v18
[(workgroup:user)@login01 src]$ cd build-v18

The documentation mentions the CMake flags what will be necessary for a build using the Intel compilers and MPI. To keep track of the steps involved, a script fragment will be created:

[(workgroup:user)@login01 build-v18]$ cat <<EOT > SWMGR-build-setup.sh
#
# v18 build
#
vpkg_require cmake/default openmpi/3.1.2:intel
( cd .. ; git checkout v18 )
cmake \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX="${TRUCHAS_TPL_PREFIX}/18" \
    -DCMAKE_C_COMPILER=icc \
    -DCMAKE_CXX_COMPILER=icpc \
    -DCMAKE_Fortran_COMPILER=ifort \
    -DMPI_C=mpicc \
    -DMPI_CXX=mpic++ \
    -DMPI_Fortran=mpifort \
    -DCMAKE_Fortran_FLAGS="-standard-semantics" \
  ..
EOT

Note the CMAKE_INSTALL_PREFIX has been set to the package's base directory augmented with the chosen version id for this build, 18.

At this point, the build can be configured:

[(workgroup:user)@login01 build-v18]$ . SWMGR-build-setup.sh
Adding package `cmake/3.21.4` to your environment
Adding dependency `intel/2018u3` to your environment
Adding dependency `libfabric/1.6.1` to your environment
Adding package `openmpi/3.1.2:intel` to your environment
   :
HEAD is now at 9201476... Merge branch 'updates' into 'master'
   :
-- The C compiler identification is Intel 18.0.3.20180410
-- The CXX compiler identification is Intel 18.0.3.20180410
-- The Fortran compiler identification is Intel 18.0.3.20180410
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/shared/intel/2018u3/compilers_and_libraries_2018.3.222/linux/bin/intel64/icc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/shared/intel/2018u3/compilers_and_libraries_2018.3.222/linux/bin/intel64/icpc - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /opt/shared/intel/2018u3/compilers_and_libraries_2018.3.222/linux/bin/intel64/ifort - skipped
-- Checking whether /opt/shared/intel/2018u3/compilers_and_libraries_2018.3.222/linux/bin/intel64/ifort supports Fortran 90
-- Checking whether /opt/shared/intel/2018u3/compilers_and_libraries_2018.3.222/linux/bin/intel64/ifort supports Fortran 90 - yes
         :
-- Build files have been written to: /work/workgroup/sw/truchas-tpl/src/build-v18

At this point the software can be built and installed according to the documentation:

[(workgroup:user)@login01 build-v18]$ make -j 4
[  3%] Creating directories for 'metis'
[  3%] Creating directories for 'chaco'
[  3%] Creating directories for 'hdf5'
[  6%] Creating directories for 'chaparral'
[  6%] Creating directories for 'hypre'
[  7%] Creating directories for 'yajl'
[ 10%] Performing download step (verify and extract) for 'chaco'
[ 10%] Performing download step (verify and extract) for 'hdf5'
      :
[(workgroup:user)@login01 build-v18]$ ls -l "$TRUCHAS_TPL_PREFIX"
total 21
drwxr-sr-x 7 user workgroup  7 Nov  4 09:55 18
drwxr-sr-x 8 user workgroup 12 Nov  4 10:53 src
[(workgroup:user)@login01 build-v18]$ ls -l "${TRUCHAS_TPL_PREFIX}/18"
total 100
drwxr-sr-x 2 user workgroup  35 Nov  4 09:55 bin
drwxr-sr-x 3 user workgroup 111 Nov  4 09:55 include
drwxr-xr-x 3 user workgroup  38 Nov  4 09:55 lib
drwxr-sr-x 4 user workgroup   7 Nov  4 09:55 lib64
drwxr-sr-x 5 user workgroup   8 Nov  4 09:55 share

The build directory for the desired commit must be created first:

[(workgroup:user)@login01 build-v18]$ cd ..
[(workgroup:user)@login01 src]$ git checkout master
Switched to branch 'master'
[(workgroup:user)@login01 src]$ VERSION_ID="$(git rev-parse --verify HEAD)"
[(workgroup:user)@login01 src]$ echo $VERSION_ID
9201476247a9ac94d6f0f4a91657fe19c9c64945
[(workgroup:user)@login01 src]$ mkdir build-${VERSION_ID}
[(workgroup:user)@login01 src]$ cd build-${VERSION_ID}

Again to keep track of the steps involved, a script fragment will be created:

[(workgroup:user)@login01 build-920…945]$ cat <<EOT > SWMGR-build-setup.sh
#
# 2021-11-04 build of HEAD of repository
#
vpkg_require cmake/default openmpi/3.1.2:intel
( cd .. ; git checkout ${VERSION_ID} )
cmake \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX="${TRUCHAS_TPL_PREFIX}/${VERSION_ID}" \
    -DCMAKE_C_COMPILER=icc \
    -DCMAKE_CXX_COMPILER=icpc \
    -DCMAKE_Fortran_COMPILER=ifort \
    -DMPI_C=mpicc \
    -DMPI_CXX=mpic++ \
    -DMPI_Fortran=mpifort \
    -DCMAKE_Fortran_FLAGS="-standard-semantics" \
  ..
EOT

The configuration and build proceeds as before, with the finished software present in a new directory:

[(workgroup:user)@login01 build-920…945]$ ls -l "$TRUCHAS_TPL_PREFIX"
total 21
drwxr-sr-x 7 user workgroup  7 Nov  4 09:55 18
drwxr-sr-x 7 user workgroup  7 Nov  4 10:30 9201476247a9ac94d6f0f4a91657fe19c9c64945
drwxr-sr-x 8 user workgroup 12 Nov  4 10:53 src
[(workgroup:user)@login01 build-920…945]$ ls -l "${TRUCHAS_TPL_PREFIX}/${VERSION_ID}"
total 100
drwxr-sr-x 2 user workgroup  35 Nov  4 10:30 bin
drwxr-sr-x 3 user workgroup 111 Nov  4 10:30 include
drwxr-xr-x 3 user workgroup  38 Nov  4 10:30 lib
drwxr-sr-x 4 user workgroup   7 Nov  4 10:30 lib64
drwxr-sr-x 5 user workgroup   8 Nov  4 10:30 share

A VALET package definition is used to encapsulate the dependencies and environment setup tasks associated with using the versions built. For a package shared among members of a workgroup, VALET searches in ${WORKDIR}/sw/valet for definition files. For the builds performed above, the package definition can be created:

truchas-tpl.vpkg_yaml

#
# VALET package definition for truchas-tpl builds
#
truchas-tpl:
    prefix: /work/workgroup/sw/truchas-tpl
    description: Truchas third-party library bundle
    url: "https://gitlab.com/truchas/truchas-tpl"
    
    default-version: "2021-11-04"
    
    versions:
        "9201476247a9ac94d6f0f4a91657fe19c9c64945":
            description: master HEAD as of 2021-11-04
            dependencies:
                - openmpi/3.1.2:intel
        "9201476":
            alias-to: "9201476247a9ac94d6f0f4a91657fe19c9c64945"
        "2021-11-04":
            alias-to: "9201476247a9ac94d6f0f4a91657fe19c9c64945"
        
        "18":
            description: commit tag v18
            dependencies:
                - openmpi/3.1.2:intel

The prefix: line in the VALET package definition will need to be changed to the proper directory for your workgroup (e.g. for workgroup it_nss it would be /work/it_nss/sw/truchas-tpl on Caviness or /lustre/it_nss/sw/truchas-tpl on DARWIN).

With that file created and installed in ${WORKDIR}/sw/valet/truchas-tpl.vpkg_yaml, all future runtime use of that package can be mediated by VALET:

[(it_nss:frey)@login01 ~]$ vpkg_versions truchas-tpl 
 
Available versions in package (* = default version):
 
[/work/it_nss/sw/valet/truchas-tpl.vpkg_yaml]
truchas-tpl                                 Truchas third-party library bundle
  18                                        commit tag v18
* 2021-11-04                                alias to truchas-tpl/9201476247a9ac94d6f0f4a91657fe19c9c64945
  9201476                                   alias to truchas-tpl/9201476247a9ac94d6f0f4a91657fe19c9c64945
  9201476247a9ac94d6f0f4a91657fe19c9c64945  master HEAD as of 2021-11-04
 
[(it_nss:frey)@login01 ~]$ vpkg_info truchas-tpl/18
[truchas-tpl/18] {
  contexts: all
  dependencies: {
    openmpi/3.1.2:intel
  }
  commit tag v18
  prefix: /work/it_nss/sw/truchas-tpl/18
  standard paths: {
    bin: /work/it_nss/sw/truchas-tpl/18/bin
    lib: /work/it_nss/sw/truchas-tpl/18/lib, /work/it_nss/sw/truchas-tpl/18/lib64
    man: /work/it_nss/sw/truchas-tpl/18/share/man
    include: /work/it_nss/sw/truchas-tpl/18/include
    pkgConfig: /work/it_nss/sw/truchas-tpl/18/lib/pkgconfig, /work/it_nss/sw/truchas-tpl/18/share/pkgconfig
  }
}
 
[(it_nss:frey)@login01 ~]$ vpkg_require truchas-tpl/18
Adding dependency `intel/2018u3` to your environment
Adding dependency `libfabric/1.6.1` to your environment
Adding dependency `openmpi/3.1.2:intel` to your environment
Adding package `truchas-tpl/18` to your environment
 
[(it_nss:frey)@login01 ~]$ which nc-config
/work/it_nss/sw/truchas-tpl/18/bin/nc-config

VALET can also add environment variables that assist in building software that depends on this package when used with vpkg_devrequire vs vpkg_require:

[(it_nss:frey)@login01 ~]$ vpkg_rollback all
[(it_nss:frey)@login01 ~]$ vpkg_devrequire truchas-tpl/18
Adding dependency `intel/2018u3` to your environment
Adding dependency `libfabric/1.6.6` to your environment
Adding dependency `openmpi/3.1.2:intel` to your environment
Adding package `truchas-tpl/18` to your environment
 
[(it_nss:frey)@login01 ~]$ echo $LDFLAGS
-L/opt/shared/libfabric/1.6.1/lib -L/work/it_nss/sw/truchas-tpl/18/lib -L/work/it_nss/sw/truchas-tpl/18/lib64
 
[(it_nss:frey)@login01 ~]$ echo $CPPFLAGS
-I/opt/shared/libfabric/1.6.1/include -I/work/it_nss/sw/truchas-tpl/18/include
 
[(it_nss:frey)@login01 ~]$ echo $CC
icc
 
[(it_nss:frey)@login01 ~]$ echo $MPICC
mpicc

Managing multiple versions of revision-controlled repositories

Setup the Hierarchy

Build a Release

Build the Current HEAD

VALET Package Definition

Configure for Development

hpc documentation