====== Managing multiple versions of revision-controlled repositories ====== Revision-controlled source code retains a historical record of the changes that a project has gone through. Milestones along this path typically represent distinct releases of the software (a.k.a. versions). Git has become an extremely popular revision-control system, partly because its embedding of all history in the local copy of the repository produces less round-trip access with the canonical repository: once a local repository has been cloned, every snapshot up to that date and time is readily available. In the more typical release strategy, a separate source code archive (e.g. a ''tar.gz'' or ''tar.bz2'' file) is present for each release containing only the source code associated with that milestone. Thus, at least one copy of each distinct version of that software package is unpacked for build purposes. There will be a great deal of repetition of information in all of those copies, which represents redundant storage that a Git repository (as a sequence of differences between snapshots) will not incur. So for a software package built from source contained in a Git repository, only one copy of the repository is ever necessary. As new snapshots become available, that single repository can be updated with a ''git pull'' against the canonical repository. In this document, the management of a Git-based software package is outlined. Workgroup storage will be used, but a path in a user's home directory would also be permissible. Common Git language references are made such as ''commit'', ''clone'', ''tag'', ''checkout'' etc. so if you are unfamiliar with Git, you may want to refer to [[https://git-scm.com/docs/git#_git_commands|Git Commands]] if you need more background information. ====== Setup the Hierarchy ====== The software package, [[https://github.com/truchas/truchas-tpl|truchas-tpl]], has tagged milestones (releases) present in its history: for example, ''v18''. Any builds of such a release will have a version identifier equivalent to the tag minus the leading "v": in the example, ''18''. Outside of the tagged revisions, a build could be performed on any commit in the history. One important commit that will likely be of interest is the HEAD of the history (the last commit made, the most recent update to the code). There are two possible versioning choices for the HEAD: the long or short commit hash, or the date on which the HEAD was used. In this example, the long commit hash will be used for the build but additional version aliases will be added to the package's VALET definition to associate the short commit hash and the date with the build. To create the package's base directory and populate a local Git repository: [(workgroup:user)@login01 ~]$ TRUCHAS_TPL_PREFIX="${WORKDIR}/sw/truchas-tpl" [(workgroup:user)@login01 ~]$ mkdir -p --mode=2775 "$TRUCHAS_TPL_PREFIX" [(workgroup:user)@login01 ~]$ cd "$TRUCHAS_TPL_PREFIX" [(workgroup:user)@login01 truchas_tpl]$ git clone https://gitlab.com/truchas/truchas-tpl.git src [(workgroup:user)@login01 truchas_tpl]$ cd src The Git repository now resides in the path ''${WORKDIR}/sw/truchas-tpl/src''. Each version of the package that is built will be installed into a directory at the same level, e.g. ''${WORKDIR}/sw/truchas-tpl/''. ====== Build a Release ====== First, examine the tagged commits available: [(workgroup:user)@login01 src]$ git tag | sort --key=1.2 --numeric-sort v1 v2 : v17 v18 A build of the ''v18'' commit will be performed. Since this package uses CMake, start by creating an out-of-tree build directory: [(workgroup:user)@login01 src]$ mkdir build-v18 [(workgroup:user)@login01 src]$ cd build-v18 The documentation mentions the CMake flags what will be necessary for a build using the Intel compilers and MPI. To keep track of the steps involved, a script fragment will be created: [(workgroup:user)@login01 build-v18]$ cat < SWMGR-build-setup.sh # # v18 build # vpkg_require cmake/default openmpi/3.1.2:intel ( cd .. ; git checkout v18 ) cmake \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_INSTALL_PREFIX="${TRUCHAS_TPL_PREFIX}/18" \ -DCMAKE_C_COMPILER=icc \ -DCMAKE_CXX_COMPILER=icpc \ -DCMAKE_Fortran_COMPILER=ifort \ -DMPI_C=mpicc \ -DMPI_CXX=mpic++ \ -DMPI_Fortran=mpifort \ -DCMAKE_Fortran_FLAGS="-standard-semantics" \ .. EOT Note the ''CMAKE_INSTALL_PREFIX'' has been set to the package's base directory augmented with the chosen version id for this build, ''18''. At this point, the build can be configured: [(workgroup:user)@login01 build-v18]$ . SWMGR-build-setup.sh Adding package `cmake/3.21.4` to your environment Adding dependency `intel/2018u3` to your environment Adding dependency `libfabric/1.6.1` to your environment Adding package `openmpi/3.1.2:intel` to your environment : HEAD is now at 9201476... Merge branch 'updates' into 'master' : -- The C compiler identification is Intel 18.0.3.20180410 -- The CXX compiler identification is Intel 18.0.3.20180410 -- The Fortran compiler identification is Intel 18.0.3.20180410 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /opt/shared/intel/2018u3/compilers_and_libraries_2018.3.222/linux/bin/intel64/icc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /opt/shared/intel/2018u3/compilers_and_libraries_2018.3.222/linux/bin/intel64/icpc - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Detecting Fortran compiler ABI info -- Detecting Fortran compiler ABI info - done -- Check for working Fortran compiler: /opt/shared/intel/2018u3/compilers_and_libraries_2018.3.222/linux/bin/intel64/ifort - skipped -- Checking whether /opt/shared/intel/2018u3/compilers_and_libraries_2018.3.222/linux/bin/intel64/ifort supports Fortran 90 -- Checking whether /opt/shared/intel/2018u3/compilers_and_libraries_2018.3.222/linux/bin/intel64/ifort supports Fortran 90 - yes : -- Build files have been written to: /work/workgroup/sw/truchas-tpl/src/build-v18 At this point the software can be built and installed according to the documentation: [(workgroup:user)@login01 build-v18]$ make -j 4 [ 3%] Creating directories for 'metis' [ 3%] Creating directories for 'chaco' [ 3%] Creating directories for 'hdf5' [ 6%] Creating directories for 'chaparral' [ 6%] Creating directories for 'hypre' [ 7%] Creating directories for 'yajl' [ 10%] Performing download step (verify and extract) for 'chaco' [ 10%] Performing download step (verify and extract) for 'hdf5' : [(workgroup:user)@login01 build-v18]$ ls -l "$TRUCHAS_TPL_PREFIX" total 21 drwxr-sr-x 7 user workgroup 7 Nov 4 09:55 18 drwxr-sr-x 8 user workgroup 12 Nov 4 10:53 src [(workgroup:user)@login01 build-v18]$ ls -l "${TRUCHAS_TPL_PREFIX}/18" total 100 drwxr-sr-x 2 user workgroup 35 Nov 4 09:55 bin drwxr-sr-x 3 user workgroup 111 Nov 4 09:55 include drwxr-xr-x 3 user workgroup 38 Nov 4 09:55 lib drwxr-sr-x 4 user workgroup 7 Nov 4 09:55 lib64 drwxr-sr-x 5 user workgroup 8 Nov 4 09:55 share ====== Build the Current HEAD ====== The build directory for the desired commit must be created first: [(workgroup:user)@login01 build-v18]$ cd .. [(workgroup:user)@login01 src]$ git checkout master Switched to branch 'master' [(workgroup:user)@login01 src]$ VERSION_ID="$(git rev-parse --verify HEAD)" [(workgroup:user)@login01 src]$ echo $VERSION_ID 9201476247a9ac94d6f0f4a91657fe19c9c64945 [(workgroup:user)@login01 src]$ mkdir build-${VERSION_ID} [(workgroup:user)@login01 src]$ cd build-${VERSION_ID} Again to keep track of the steps involved, a script fragment will be created: [(workgroup:user)@login01 build-920…945]$ cat < SWMGR-build-setup.sh # # 2021-11-04 build of HEAD of repository # vpkg_require cmake/default openmpi/3.1.2:intel ( cd .. ; git checkout ${VERSION_ID} ) cmake \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_INSTALL_PREFIX="${TRUCHAS_TPL_PREFIX}/${VERSION_ID}" \ -DCMAKE_C_COMPILER=icc \ -DCMAKE_CXX_COMPILER=icpc \ -DCMAKE_Fortran_COMPILER=ifort \ -DMPI_C=mpicc \ -DMPI_CXX=mpic++ \ -DMPI_Fortran=mpifort \ -DCMAKE_Fortran_FLAGS="-standard-semantics" \ .. EOT The configuration and build proceeds as before, with the finished software present in a new directory: [(workgroup:user)@login01 build-920…945]$ ls -l "$TRUCHAS_TPL_PREFIX" total 21 drwxr-sr-x 7 user workgroup 7 Nov 4 09:55 18 drwxr-sr-x 7 user workgroup 7 Nov 4 10:30 9201476247a9ac94d6f0f4a91657fe19c9c64945 drwxr-sr-x 8 user workgroup 12 Nov 4 10:53 src [(workgroup:user)@login01 build-920…945]$ ls -l "${TRUCHAS_TPL_PREFIX}/${VERSION_ID}" total 100 drwxr-sr-x 2 user workgroup 35 Nov 4 10:30 bin drwxr-sr-x 3 user workgroup 111 Nov 4 10:30 include drwxr-xr-x 3 user workgroup 38 Nov 4 10:30 lib drwxr-sr-x 4 user workgroup 7 Nov 4 10:30 lib64 drwxr-sr-x 5 user workgroup 8 Nov 4 10:30 share ====== VALET Package Definition ====== A VALET package definition is used to encapsulate the dependencies and environment setup tasks associated with using the versions built. For a package shared among members of a workgroup, VALET searches in ''${WORKDIR}/sw/valet'' for definition files. For the builds performed above, the package definition can be created: # # VALET package definition for truchas-tpl builds # truchas-tpl: prefix: /work/workgroup/sw/truchas-tpl description: Truchas third-party library bundle url: "https://gitlab.com/truchas/truchas-tpl" default-version: "2021-11-04" versions: "9201476247a9ac94d6f0f4a91657fe19c9c64945": description: master HEAD as of 2021-11-04 dependencies: - openmpi/3.1.2:intel "9201476": alias-to: "9201476247a9ac94d6f0f4a91657fe19c9c64945" "2021-11-04": alias-to: "9201476247a9ac94d6f0f4a91657fe19c9c64945" "18": description: commit tag v18 dependencies: - openmpi/3.1.2:intel The ''prefix:'' line in the VALET package definition will need to be changed to the proper directory for your workgroup (e.g. for workgroup ''it_nss'' it would be ''/work/it_nss/sw/truchas-tpl'' on Caviness or ''/lustre/it_nss/sw/truchas-tpl'' on DARWIN). With that file created and installed in ''${WORKDIR}/sw/valet/truchas-tpl.vpkg_yaml'', all future runtime use of that package can be mediated by VALET: [(it_nss:frey)@login01 ~]$ vpkg_versions truchas-tpl Available versions in package (* = default version): [/work/it_nss/sw/valet/truchas-tpl.vpkg_yaml] truchas-tpl Truchas third-party library bundle 18 commit tag v18 * 2021-11-04 alias to truchas-tpl/9201476247a9ac94d6f0f4a91657fe19c9c64945 9201476 alias to truchas-tpl/9201476247a9ac94d6f0f4a91657fe19c9c64945 9201476247a9ac94d6f0f4a91657fe19c9c64945 master HEAD as of 2021-11-04 [(it_nss:frey)@login01 ~]$ vpkg_info truchas-tpl/18 [truchas-tpl/18] { contexts: all dependencies: { openmpi/3.1.2:intel } commit tag v18 prefix: /work/it_nss/sw/truchas-tpl/18 standard paths: { bin: /work/it_nss/sw/truchas-tpl/18/bin lib: /work/it_nss/sw/truchas-tpl/18/lib, /work/it_nss/sw/truchas-tpl/18/lib64 man: /work/it_nss/sw/truchas-tpl/18/share/man include: /work/it_nss/sw/truchas-tpl/18/include pkgConfig: /work/it_nss/sw/truchas-tpl/18/lib/pkgconfig, /work/it_nss/sw/truchas-tpl/18/share/pkgconfig } } [(it_nss:frey)@login01 ~]$ vpkg_require truchas-tpl/18 Adding dependency `intel/2018u3` to your environment Adding dependency `libfabric/1.6.1` to your environment Adding dependency `openmpi/3.1.2:intel` to your environment Adding package `truchas-tpl/18` to your environment [(it_nss:frey)@login01 ~]$ which nc-config /work/it_nss/sw/truchas-tpl/18/bin/nc-config ===== Configure for Development ===== VALET can also add environment variables that assist in building software that depends on this package when used with ''vpkg_devrequire'' vs ''vpkg_require'': [(it_nss:frey)@login01 ~]$ vpkg_rollback all [(it_nss:frey)@login01 ~]$ vpkg_devrequire truchas-tpl/18 Adding dependency `intel/2018u3` to your environment Adding dependency `libfabric/1.6.6` to your environment Adding dependency `openmpi/3.1.2:intel` to your environment Adding package `truchas-tpl/18` to your environment [(it_nss:frey)@login01 ~]$ echo $LDFLAGS -L/opt/shared/libfabric/1.6.1/lib -L/work/it_nss/sw/truchas-tpl/18/lib -L/work/it_nss/sw/truchas-tpl/18/lib64 [(it_nss:frey)@login01 ~]$ echo $CPPFLAGS -I/opt/shared/libfabric/1.6.1/include -I/work/it_nss/sw/truchas-tpl/18/include [(it_nss:frey)@login01 ~]$ echo $CC icc [(it_nss:frey)@login01 ~]$ echo $MPICC mpicc