technical:recipes:git-cmake-valet-package

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
technical:recipes:git-cmake-valet-package [2021-11-04 11:58] – [Managing multiple versions of revision-controlled repositories] anitatechnical:recipes:git-cmake-valet-package [2021-11-18 17:58] (current) – frey
Line 1: Line 1:
 +====== Managing multiple versions of revision-controlled repositories ======
  
 +Revision-controlled source code retains a historical record of the changes that a project has gone through.  Milestones along this path typically represent distinct releases of the software (a.k.a. versions).  Git has become an extremely popular revision-control system, partly because its embedding of all history in the local copy of the repository produces less round-trip access with the canonical repository:  once  a local repository has been cloned, every snapshot up to that date and time is readily available.
 +
 +In the more typical release strategy, a separate source code archive (e.g. a ''tar.gz'' or ''tar.bz2'' file) is present for each release containing only the source code associated with that milestone.  Thus, at least one copy of each distinct version of that software package is unpacked for build purposes.  There will be a great deal of repetition of information in all of those copies, which represents redundant storage that a Git repository (as a sequence of differences between snapshots) will not incur.  So for a software package built from source contained in a Git repository, only one copy of the repository is ever necessary.  As new snapshots become available, that single repository can be updated with a ''git pull'' against the canonical repository.
 +
 +In this document, the management of a Git-based software package is outlined.  Workgroup storage will be used, but a path in a user's home directory would also be permissible. Common Git language references are made such as ''commit'', ''clone'', ''tag'', ''checkout'' etc. so if you are unfamiliar with Git, you may want to refer to [[https://git-scm.com/docs/git#_git_commands|Git Commands]] if you need more background information.
 +
 +====== Setup the Hierarchy ======
 +
 +The software package, [[https://github.com/truchas/truchas-tpl|truchas-tpl]], has tagged milestones (releases) present in its history:  for example, ''v18''.  Any builds of such a release will have a version identifier equivalent to the tag minus the leading "v":  in the example, ''18''.
 +
 +Outside of the tagged revisions, a build could be performed on any commit in the history.  One important commit that will likely be of interest is the HEAD of the history (the last commit made, the most recent update to the code).  There are two possible versioning choices for the HEAD:  the long or short commit hash, or the date on which the HEAD was used.  In this example, the long commit hash will be used for the build but additional version aliases will be added to the package's VALET definition to associate the short commit hash and the date with the build.
 +
 +To create the package's base directory and populate a local Git repository:
 +
 +<code bash>
 +[(workgroup:user)@login01 ~]$ TRUCHAS_TPL_PREFIX="${WORKDIR}/sw/truchas-tpl"
 +[(workgroup:user)@login01 ~]$ mkdir -p --mode=2775 "$TRUCHAS_TPL_PREFIX"
 +[(workgroup:user)@login01 ~]$ cd "$TRUCHAS_TPL_PREFIX"
 +[(workgroup:user)@login01 truchas_tpl]$ git clone https://gitlab.com/truchas/truchas-tpl.git src
 +[(workgroup:user)@login01 truchas_tpl]$ cd src
 +</code>
 +
 +The Git repository now resides in the path ''${WORKDIR}/sw/truchas-tpl/src''.  Each version of the package that is built will be installed into a directory at the same level, e.g. ''${WORKDIR}/sw/truchas-tpl/<version>''.
 +
 +====== Build a Release ======
 +
 +First, examine the tagged commits available:
 +
 +<code bash>
 +[(workgroup:user)@login01 src]$ git tag | sort --key=1.2 --numeric-sort
 +v1
 +v2
 + :
 +v17
 +v18
 +</code>
 +
 +A build of the ''v18'' commit will be performed.  Since this package uses CMake, start by creating an out-of-tree build directory:
 +
 +<code bash>
 +[(workgroup:user)@login01 src]$ mkdir build-v18
 +[(workgroup:user)@login01 src]$ cd build-v18
 +</code>
 +
 +The documentation mentions the CMake flags what will be necessary for a build using the Intel compilers and MPI.  To keep track of the steps involved, a script fragment will be created:
 +
 +<code bash>
 +[(workgroup:user)@login01 build-v18]$ cat <<EOT > SWMGR-build-setup.sh
 +#
 +# v18 build
 +#
 +vpkg_require cmake/default openmpi/3.1.2:intel
 +( cd .. ; git checkout v18 )
 +cmake \
 +    -DCMAKE_BUILD_TYPE=Release \
 +    -DCMAKE_INSTALL_PREFIX="${TRUCHAS_TPL_PREFIX}/18" \
 +    -DCMAKE_C_COMPILER=icc \
 +    -DCMAKE_CXX_COMPILER=icpc \
 +    -DCMAKE_Fortran_COMPILER=ifort \
 +    -DMPI_C=mpicc \
 +    -DMPI_CXX=mpic++ \
 +    -DMPI_Fortran=mpifort \
 +    -DCMAKE_Fortran_FLAGS="-standard-semantics" \
 +  ..
 +EOT
 +</code>
 +
 +Note the ''CMAKE_INSTALL_PREFIX'' has been set to the package's base directory augmented with the chosen version id for this build, ''18''.
 +
 +At this point, the build can be configured:
 +
 +<code bash>
 +[(workgroup:user)@login01 build-v18]$ . SWMGR-build-setup.sh
 +Adding package `cmake/3.21.4` to your environment
 +Adding dependency `intel/2018u3` to your environment
 +Adding dependency `libfabric/1.6.1` to your environment
 +Adding package `openmpi/3.1.2:intel` to your environment
 +   :
 +HEAD is now at 9201476... Merge branch 'updates' into 'master'
 +   :
 +-- The C compiler identification is Intel 18.0.3.20180410
 +-- The CXX compiler identification is Intel 18.0.3.20180410
 +-- The Fortran compiler identification is Intel 18.0.3.20180410
 +-- Detecting C compiler ABI info
 +-- Detecting C compiler ABI info - done
 +-- Check for working C compiler: /opt/shared/intel/2018u3/compilers_and_libraries_2018.3.222/linux/bin/intel64/icc - skipped
 +-- Detecting C compile features
 +-- Detecting C compile features - done
 +-- Detecting CXX compiler ABI info
 +-- Detecting CXX compiler ABI info - done
 +-- Check for working CXX compiler: /opt/shared/intel/2018u3/compilers_and_libraries_2018.3.222/linux/bin/intel64/icpc - skipped
 +-- Detecting CXX compile features
 +-- Detecting CXX compile features - done
 +-- Detecting Fortran compiler ABI info
 +-- Detecting Fortran compiler ABI info - done
 +-- Check for working Fortran compiler: /opt/shared/intel/2018u3/compilers_and_libraries_2018.3.222/linux/bin/intel64/ifort - skipped
 +-- Checking whether /opt/shared/intel/2018u3/compilers_and_libraries_2018.3.222/linux/bin/intel64/ifort supports Fortran 90
 +-- Checking whether /opt/shared/intel/2018u3/compilers_and_libraries_2018.3.222/linux/bin/intel64/ifort supports Fortran 90 - yes
 +         :
 +-- Build files have been written to: /work/workgroup/sw/truchas-tpl/src/build-v18
 +</code>
 +
 +At this point the software can be built and installed according to the documentation:
 +
 +<code bash>
 +[(workgroup:user)@login01 build-v18]$ make -j 4
 +[  3%] Creating directories for 'metis'
 +[  3%] Creating directories for 'chaco'
 +[  3%] Creating directories for 'hdf5'
 +[  6%] Creating directories for 'chaparral'
 +[  6%] Creating directories for 'hypre'
 +[  7%] Creating directories for 'yajl'
 +[ 10%] Performing download step (verify and extract) for 'chaco'
 +[ 10%] Performing download step (verify and extract) for 'hdf5'
 +      :
 +[(workgroup:user)@login01 build-v18]$ ls -l "$TRUCHAS_TPL_PREFIX"
 +total 21
 +drwxr-sr-x 7 user workgroup  7 Nov  4 09:55 18
 +drwxr-sr-x 8 user workgroup 12 Nov  4 10:53 src
 +[(workgroup:user)@login01 build-v18]$ ls -l "${TRUCHAS_TPL_PREFIX}/18"
 +total 100
 +drwxr-sr-x 2 user workgroup  35 Nov  4 09:55 bin
 +drwxr-sr-x 3 user workgroup 111 Nov  4 09:55 include
 +drwxr-xr-x 3 user workgroup  38 Nov  4 09:55 lib
 +drwxr-sr-x 4 user workgroup   7 Nov  4 09:55 lib64
 +drwxr-sr-x 5 user workgroup   8 Nov  4 09:55 share
 +</code>
 +
 +====== Build the Current HEAD ======
 +
 +The build directory for the desired commit must be created first:
 +
 +<code bash>
 +[(workgroup:user)@login01 build-v18]$ cd ..
 +[(workgroup:user)@login01 src]$ git checkout master
 +Switched to branch 'master'
 +[(workgroup:user)@login01 src]$ VERSION_ID="$(git rev-parse --verify HEAD)"
 +[(workgroup:user)@login01 src]$ echo $VERSION_ID
 +9201476247a9ac94d6f0f4a91657fe19c9c64945
 +[(workgroup:user)@login01 src]$ mkdir build-${VERSION_ID}
 +[(workgroup:user)@login01 src]$ cd build-${VERSION_ID}
 +</code>
 +
 +Again to keep track of the steps involved, a script fragment will be created:
 +
 +<code bash>
 +[(workgroup:user)@login01 build-920…945]$ cat <<EOT > SWMGR-build-setup.sh
 +#
 +# 2021-11-04 build of HEAD of repository
 +#
 +vpkg_require cmake/default openmpi/3.1.2:intel
 +( cd .. ; git checkout ${VERSION_ID} )
 +cmake \
 +    -DCMAKE_BUILD_TYPE=Release \
 +    -DCMAKE_INSTALL_PREFIX="${TRUCHAS_TPL_PREFIX}/${VERSION_ID}" \
 +    -DCMAKE_C_COMPILER=icc \
 +    -DCMAKE_CXX_COMPILER=icpc \
 +    -DCMAKE_Fortran_COMPILER=ifort \
 +    -DMPI_C=mpicc \
 +    -DMPI_CXX=mpic++ \
 +    -DMPI_Fortran=mpifort \
 +    -DCMAKE_Fortran_FLAGS="-standard-semantics" \
 +  ..
 +EOT
 +</code>
 +
 +The configuration and build proceeds as before, with the finished software present in a new directory:
 +
 +<code bash>
 +[(workgroup:user)@login01 build-920…945]$ ls -l "$TRUCHAS_TPL_PREFIX"
 +total 21
 +drwxr-sr-x 7 user workgroup  7 Nov  4 09:55 18
 +drwxr-sr-x 7 user workgroup  7 Nov  4 10:30 9201476247a9ac94d6f0f4a91657fe19c9c64945
 +drwxr-sr-x 8 user workgroup 12 Nov  4 10:53 src
 +[(workgroup:user)@login01 build-920…945]$ ls -l "${TRUCHAS_TPL_PREFIX}/${VERSION_ID}"
 +total 100
 +drwxr-sr-x 2 user workgroup  35 Nov  4 10:30 bin
 +drwxr-sr-x 3 user workgroup 111 Nov  4 10:30 include
 +drwxr-xr-x 3 user workgroup  38 Nov  4 10:30 lib
 +drwxr-sr-x 4 user workgroup   7 Nov  4 10:30 lib64
 +drwxr-sr-x 5 user workgroup   8 Nov  4 10:30 share
 +</code>
 +
 +====== VALET Package Definition ======
 +
 +A VALET package definition is used to encapsulate the dependencies and environment setup tasks associated with using the versions built.  For a package shared among members of a workgroup, VALET searches in ''${WORKDIR}/sw/valet'' for definition files.  For the builds performed above, the package definition can be created:
 +
 +<file yaml truchas-tpl.vpkg_yaml>
 +#
 +# VALET package definition for truchas-tpl builds
 +#
 +truchas-tpl:
 +    prefix: /work/workgroup/sw/truchas-tpl
 +    description: Truchas third-party library bundle
 +    url: "https://gitlab.com/truchas/truchas-tpl"
 +    
 +    default-version: "2021-11-04"
 +    
 +    versions:
 +        "9201476247a9ac94d6f0f4a91657fe19c9c64945":
 +            description: master HEAD as of 2021-11-04
 +            dependencies:
 +                - openmpi/3.1.2:intel
 +        "9201476":
 +            alias-to: "9201476247a9ac94d6f0f4a91657fe19c9c64945"
 +        "2021-11-04":
 +            alias-to: "9201476247a9ac94d6f0f4a91657fe19c9c64945"
 +        
 +        "18":
 +            description: commit tag v18
 +            dependencies:
 +                - openmpi/3.1.2:intel
 +
 +</file>
 +
 +<note important>The ''prefix:'' line in the VALET package definition will need to be changed to the proper directory for your workgroup (e.g. for workgroup ''it_nss'' it would be ''/work/it_nss/sw/truchas-tpl'' on Caviness or ''/lustre/it_nss/sw/truchas-tpl'' on DARWIN).</note>
 +
 +With that file created and installed in ''${WORKDIR}/sw/valet/truchas-tpl.vpkg_yaml'', all future runtime use of that package can be mediated by VALET:
 +
 +<code bash>
 +[(it_nss:frey)@login01 ~]$ vpkg_versions truchas-tpl 
 +
 +Available versions in package (* = default version):
 +
 +[/work/it_nss/sw/valet/truchas-tpl.vpkg_yaml]
 +truchas-tpl                                 Truchas third-party library bundle
 +  18                                        commit tag v18
 +* 2021-11-04                                alias to truchas-tpl/9201476247a9ac94d6f0f4a91657fe19c9c64945
 +  9201476                                   alias to truchas-tpl/9201476247a9ac94d6f0f4a91657fe19c9c64945
 +  9201476247a9ac94d6f0f4a91657fe19c9c64945  master HEAD as of 2021-11-04
 +
 +[(it_nss:frey)@login01 ~]$ vpkg_info truchas-tpl/18
 +[truchas-tpl/18] {
 +  contexts: all
 +  dependencies: {
 +    openmpi/3.1.2:intel
 +  }
 +  commit tag v18
 +  prefix: /work/it_nss/sw/truchas-tpl/18
 +  standard paths: {
 +    bin: /work/it_nss/sw/truchas-tpl/18/bin
 +    lib: /work/it_nss/sw/truchas-tpl/18/lib, /work/it_nss/sw/truchas-tpl/18/lib64
 +    man: /work/it_nss/sw/truchas-tpl/18/share/man
 +    include: /work/it_nss/sw/truchas-tpl/18/include
 +    pkgConfig: /work/it_nss/sw/truchas-tpl/18/lib/pkgconfig, /work/it_nss/sw/truchas-tpl/18/share/pkgconfig
 +  }
 +}
 +
 +[(it_nss:frey)@login01 ~]$ vpkg_require truchas-tpl/18
 +Adding dependency `intel/2018u3` to your environment
 +Adding dependency `libfabric/1.6.1` to your environment
 +Adding dependency `openmpi/3.1.2:intel` to your environment
 +Adding package `truchas-tpl/18` to your environment
 +
 +[(it_nss:frey)@login01 ~]$ which nc-config
 +/work/it_nss/sw/truchas-tpl/18/bin/nc-config
 +</code>
 +
 +===== Configure for Development =====
 +
 +VALET can also add environment variables that assist in building software that depends on this package when used with ''vpkg_devrequire'' vs ''vpkg_require'':
 +
 +<code bash>
 +[(it_nss:frey)@login01 ~]$ vpkg_rollback all
 +[(it_nss:frey)@login01 ~]$ vpkg_devrequire truchas-tpl/18
 +Adding dependency `intel/2018u3` to your environment
 +Adding dependency `libfabric/1.6.6` to your environment
 +Adding dependency `openmpi/3.1.2:intel` to your environment
 +Adding package `truchas-tpl/18` to your environment
 +
 +[(it_nss:frey)@login01 ~]$ echo $LDFLAGS
 +-L/opt/shared/libfabric/1.6.1/lib -L/work/it_nss/sw/truchas-tpl/18/lib -L/work/it_nss/sw/truchas-tpl/18/lib64
 +
 +[(it_nss:frey)@login01 ~]$ echo $CPPFLAGS
 +-I/opt/shared/libfabric/1.6.1/include -I/work/it_nss/sw/truchas-tpl/18/include
 +
 +[(it_nss:frey)@login01 ~]$ echo $CC
 +icc
 +
 +[(it_nss:frey)@login01 ~]$ echo $MPICC
 +mpicc
 +</code>