====== Building Gromacs-Plumed on Caviness/DARWIN ======

The build procedure outlined herein uses Open MPI on top of the Intel compiler suite.

===== Directory Preparation =====

To begin, choose a directory in which the Gromacs_Plumed version(s) will be built and installed.  To build in your home directory, for example:
<code bash>
[user@login00.darwin ~]$ GP_BASEDIR=~/sw/Gromacs_Plumed
[user@login00.darwin ~]$ G_BASEDIR="${GP_BASEDIR}/Gromacs"
[user@login00.darwin ~]$ P_BASEDIR="${GP_BASEDIR}/Plumed"
[user@login00.darwin ~]$ GP_BASEDIR_PRIVS=0700
</code>
If you are managing Gromacs_Plumed software for your entire workgroup, you could instead use
<code bash>
[user@login00.darwin ~]$ GP_BASEDIR="${WORKDIR}/sw/Gromacs_Plumed"
[user@login00.darwin ~]$ G_BASEDIR="${GP_BASEDIR}/Gromacs"
[user@login00.darwin ~]$ P_BASEDIR="${GP_BASEDIR}/Plumed"
[user@login00.darwin ~]$ GP_BASEDIR_PRIVS=2770
</code>


If the directory hierarchy does not yet exist, it can be setup using
<code bash>
[user@login00.darwin ~]$ mkdir -p -m $GP_BASEDIR_PRIVS "${GP_BASEDIR}"
[user@login00.darwin ~]$ mkdir -p -m $GP_BASEDIR_PRIVS "${G_BASEDIR}"
[user@login00.darwin ~]$ mkdir -p -m $GP_BASEDIR_PRIVS "${P_BASEDIR}"
</code>

===== Source Preparation: Plumed =====

In this example version 2.8.3 of Plumed will be built. Our standard recipes for Plumed will entail use of the Intel compilers, the MKL for BLAS/LAPACK/FFTW/ScaLAPACK/BLACS, and Open MPI for parallelism.

Download and unpack "plumed-2.8.3.tgz" and create a directory to hold our base build of plumed-2.8.3, naming it with the version identifier:  ''v2.8.3''.  The source is then unpacked therein:
<code bash>
[user@login00.darwin ~]$ cd $P_BASEDIR
[user@login00.darwin Plumed]$ wget https://github.com/plumed/plumed2/releases/download/v2.8.3/plumed-2.8.3.tgz
[user@login00.darwin Plumed]$ tar -xvzf plumed-2.8.3.tgz
</code>

===== Compilation and Installation: Plumed=====

We use configure for compilation and installation on "v2.8.3" directory.
<code bash>
[user@login00.darwin ~]$ P_INSTALL_PREFIX="${P_BASEDIR}/v2.8.3"
[user@login00.darwin ~]$ export P_INSTALL_PREFIX
[user@login00.darwin ~]$ P_SRC_PREFIX="${P_BASEDIR}/plumed-2.8.3"
[user@login00.darwin ~]$ mkdir -m $GP_BASEDIR_PRIVS "$P_INSTALL_PREFIX"
[user@login00.darwin ~]$ cd "$P_SRC_PREFIX"
[user@login00.darwin plumed-2.8.3]$ vpkg_devrequire openmpi/4.1.5:intel-oneapi-2023
[user@login00.darwin plumed-2.8.3]$ ./configure --prefix=$P_INSTALL_PREFIX  CC=mpicc CXX=mpicxx FC=mpifort  LDFLAGS=-qmkl=sequential 
[user@login00.darwin plumed-2.8.3]$ make
[user@login00.darwin plumed-2.8.3]$ make install
</code>
Our current working directory is now the source directory.
Here, we create ''sourceme.sh'' :
<file make sourceme.sh>
TPATH="${P_INSTALL_PREFIX}"
export PATH="$TPATH/bin/:$PATH"
export LIBRARY_PATH="$TPATH/lib/:$LIBRARY_PATH"
export LD_LIBRARY_PATH="$TPATH/lib/:$LD_LIBRARY_PATH"
export PLUMED_KERNEL="$TPATH/lib/libplumedKernel.so"
export PLUMED_VIMPATH="$TPATH/lib/plumed/vim/"
export PYTHONPATH="$TPATH/lib/python/:$PYTHONPATH"

export CPATH="$TPATH/include/:$CPATH”
export INCLUDE="$TPATH/include/:$INCLUDE”
</file>


Executing the script.
<code bash>
[user@login00.darwin plumed-2.8.3]$ source sourceme.sh
</code> 

===== Source Preparation: Gromacs =====

In this example version 2022.5 of Gromacs will be built. Our standard recipes for Gromacs will entail use of the Intel compilers, the MKL for BLAS/LAPACK/FFTW/ScaLAPACK/BLACS, and Open MPI for parallelism.

Download and unpack "gromacs-2022.5.tar.gz" and create a directory to hold our base build of gromacs-2022.5, naming it with the version identifier:  ''v2022.5''.  The source is then unpacked therein:
<code bash>
[user@login00.darwin plumed-2.8.3]$ cd $G_BASEDIR
[user@login00.darwin Gromacs]$ wget https://ftp.gromacs.org/gromacs/gromacs-2022.5.tar.gz
[user@login00.darwin Gromacs]$ tar -xvzf gromacs-2022.5.tar.gz
</code>

===== Compilation and Installation: Plumed-patched Gromacs =====

We use cmake for compilation and installation on Plumed-patched Gromacs "v2022.5" directory.
<code bash>
[user@login00.darwin Gromacs]$ G_INSTALL_PREFIX="${G_BASEDIR}/v2022.5"
[user@login00.darwin Gromacs]$ export G_INSTALL_PREFIX
[user@login00.darwin Gromacs]$ G_SRC_PREFIX="${G_BASEDIR}/gromacs-2022.5"
[user@login00.darwin Gromacs]$ mkdir -m $GP_BASEDIR_PRIVS "$G_INSTALL_PREFIX"
[user@login00.darwin Gromacs]$ G_BUILDDIR="${G_SRC_PREFIX}/build"
[user@login00.darwin Gromacs]$ mkdir -m $GP_BASEDIR_PRIVS "$G_BUILDDIR"
[user@login00.darwin Gromacs]$ cd "$G_SRC_PREFIX"
</code>
Hear, we patch Plumed on Gromacs. 
<code bash>
[user@login00.darwin gromacs-2022.5]$ plumed patch -p --static
</code>

Select 4:
<code bash>
1) gromacs-2019.6     5) gromacs-4.5.7	   9) qespresso-5.0.2
2) gromacs-2020.7     6) namd-2.12	  10) qespresso-6.2
3) gromacs-2021.7     7) namd-2.13
4) gromacs-2022.5     8) namd-2.14
</code>

<code bash>
[user@login00.darwin gromacs-2022.5]$ cd "$G_BUILDDIR"
</code> 
Our current working directory is now the //build root//.
Here, we create ''SWMGR-build.sh'' :
<file make SWMGR-build.sh.darwin>
#!/bin/bash -l

vpkg_require cmake/default openmpi/4.1.5:intel-oneapi-2023

PREFIX="${G_INSTALL_PREFIX}"

CPU_GMX_SIMD=AVX2_256

BASE_BUILD_FLAGS="
    -DCMAKE_BUILD_TYPE=Release
    -DGMX_SIMD=${CPU_GMX_SIMD}
    -DGMX_FFT_LIBRARY=mkl
    -DMKL_MPI=openmpi
    -DBUILD_SHARED_LIBS=OFF
    -DGMX_PREFER_STATIC_LIBS=ON
"
# BASE_BUILD_FLAGS="$BASE_BUILD_FLAGS -DGMX_GPU=SYCL"
#    -DMKL_INCLUDE_DIR=$MKLROOT/include
#    -DMKL_LIBRARIES=-mkl=sequential
#"

build_variant()
{
    local DBL=$1 MPI=$2
    local build_dir=""
    local build_flags=""

    if [ $DBL -ne 0 ]; then
        build_dir="double"
	build_flags="${build_flags} -DGMX_DOUBLE=on"
    else
        build_dir="single"
        build_flags="${build_flags} -DGMX_DOUBLE=off"
    fi
    if [ $MPI -eq 0 ]; then
        build_dir="${build_dir}-thread"
        build_flags="${build_flags} -DGMX_MPI=off -DGMX_THREAD_MPI=on -DCMAKE_DISABLE_FIND_PACKAGE_MPI=on"
        CC=icx CXX=icpx FC=ifx
    else
        build_dir="${build_dir}-mpi"
        build_flags="${build_flags} -DGMX_MPI=on -DGMX_THREAD_MPI=off"
	CC=mpicc CXX=mpicxx FC=mpifort
    fi
    [ -d "build-${build_dir}" ] && rm -rf "build-${build_dir}"
    mkdir "build-${build_dir}" ; pushd "build-${build_dir}"
    CC=$CC CXX=$CXX FC=$FC \
        cmake \
	    -DCMAKE_INSTALL_PREFIX="${PREFIX}/${build_dir}" \
            $BASE_BUILD_FLAGS $build_flags \
	../..
    if [ $? -eq 0 ]; then
        make -j 20
        if [ $? -eq 0 ]; then
            make install
        fi
    fi
    popd
}

##
## SYCL only supports single precision.
##
#             DOUBLE?    MPI?
#build_variant 0          0
#build_variant 1          0
#build_variant 0          1
build_variant 1          1

</file>

<file make SWMGR-build.sh.caviness>
#!/bin/bash -l

vpkg_require cmake/default openmpi/4.1.4:intel-oneapi-2023

PREFIX="${G_INSTALL_PREFIX}"

CPU_GMX_SIMD=AVX2_256

BASE_BUILD_FLAGS="
    -DCMAKE_BUILD_TYPE=Release
    -DGMX_SIMD=${CPU_GMX_SIMD}
    -DGMX_FFT_LIBRARY=mkl
    -DMKL_MPI=openmpi
    -DBUILD_SHARED_LIBS=OFF
    -DGMX_PREFER_STATIC_LIBS=ON
"
# BASE_BUILD_FLAGS="$BASE_BUILD_FLAGS -DGMX_GPU=SYCL"
#    -DMKL_INCLUDE_DIR=$MKLROOT/include
#    -DMKL_LIBRARIES=-mkl=sequential
#"

build_variant()
{
    local DBL=$1 MPI=$2
    local build_dir=""
    local build_flags=""

    if [ $DBL -ne 0 ]; then
        build_dir="double"
	build_flags="${build_flags} -DGMX_DOUBLE=on"
    else
        build_dir="single"
        build_flags="${build_flags} -DGMX_DOUBLE=off"
    fi
    if [ $MPI -eq 0 ]; then
        build_dir="${build_dir}-thread"
        build_flags="${build_flags} -DGMX_MPI=off -DGMX_THREAD_MPI=on -DCMAKE_DISABLE_FIND_PACKAGE_MPI=on"
        CC=icx CXX=icpx FC=ifx
    else
        build_dir="${build_dir}-mpi"
        build_flags="${build_flags} -DGMX_MPI=on -DGMX_THREAD_MPI=off"
	CC=mpicc CXX=mpicxx FC=mpifort
    fi
    [ -d "build-${build_dir}" ] && rm -rf "build-${build_dir}"
    mkdir "build-${build_dir}" ; pushd "build-${build_dir}"
    CC=$CC CXX=$CXX FC=$FC \
        cmake \
	    -DCMAKE_INSTALL_PREFIX="${PREFIX}/${build_dir}" \
            $BASE_BUILD_FLAGS $build_flags \
	../..
    if [ $? -eq 0 ]; then
        make -j 20
        if [ $? -eq 0 ]; then
            make install
        fi
    fi
    popd
}

##
## SYCL only supports single precision.
##
#             DOUBLE?    MPI?
#build_variant 0          0
#build_variant 1          0
#build_variant 0          1
build_variant 1          1

</file>

Executing the script compile and install plumed-patched gromacs.
<code bash>
[user@login00.darwin build]$ ./SWMGR-build.sh
</code> 

===== VALET Package Definition =====

With this version of Gromacs_Plumed built, the remaining step is to leverage VALET for setup of the runtime environment when you use the software.  VALET automatically recognizes the standard directory layout, so configuring versions/variants of ''Gromacs_Plumed'' is very straightforward.  First, note your installation path:
<code bash>
[user@login00.darwin build]$ vpkg_rollback all
[user@login00.darwin build]$ cd
[user@login00.darwin ~]$ echo $GP_BASEDIR
/home/user/sw/Gromacs_Plumed
</code>
Since this build was done in the user's home directory, they were personal copies of the software and should use a //VALET package definition file// stored in ''~/.valet''
<code bash>
[user@login00.darwin ~]$ VALET_PKG_DIR=~/.valet ; VALET_PKG_DIR_MODE=0700
</code>
versus an installation made for an entire workgroup, which would store the VALET package definition files in ''$WORKDIR/sw/valet''
<code bash>
[user@login00.darwin ~]$ VALET_PKG_DIR="$WORKDIR/sw/valet" ; VALET_PKG_DIR_MODE=2770
</code>
Whichever scheme is in-use, ensure the directory exists:
<code bash>
[user@login00.darwin ~]$ mkdir -p --mode=$VALET_PKG_DIR_MODE "$VALET_PKG_DIR"
</code>

VALET allows package definitions in a variety of formats (XML, JSON, YAML) but YAML tends to be the simplest format so we will use it here.

==== Package section ====

The //package section// of the definition file includes items that apply to all versions/variants of the software:
<code yaml>
Gromacs_Plumed:
    prefix: /home/user/sw/Gromacs_Plumed/Gromacs
    description: Gromacs_Plumed
    url: "https://www.gromacs.org/"
</code>
The //package identifier// is the top-level key in the document — ''Gromacs_Plumed'' — and the value of ''$GP_BASEDIR'' is the value of the ''prefix'' key in this section.  The URL and description are information taken from the official Gromacs web site.

==== Versions ====

The ''versions'' key is used to provide a list of the versions/variants of the software:
<code yaml>
gromacs-plumed:
    prefix: /home/user/sw/Gromacs_Plumed/Gromacs
    description: Gromacs_Plumed
    url: "https://www.gromacs.org/"
    
    default-version: "v2022.5"
    
    versions:
        "v2022.5":
            prefix: "v2022.5/double-mpi"
            description:  Gromacs patched with Plumed v2.8.3
            dependencies:
                 - openmpi/4.1.5:intel-oneapi-2023
               
</code>

<WRAP center round tip 80%>
The version identifier ''v7.3'' is inferred to be the path prefix to the version in question here.  The package's prefix (''/home/user/sw/Gromacs_Plumed'') with the version identifier appended (''/home/user/sw/Gromacs_Plumed/Gromacs/v2022.5'') is implicit.

The implicit behavior is overridden by providing a ''prefix'' key in the version definition:  a relative path is appended to the package's prefix, an absolute path is used as-is.
</WRAP>

It is a good idea to specify which version definition should act as the default.  This yields the following package definition file

<file yaml gromacs-plumed.vpkg_yaml.darwin>
gromacs-plumed:
    prefix: /home/user/sw/Gromacs_Plumed/Gromacs
    description: Gromacs_Plumed
    url: "https://www.gromacs.org/"
    
    default-version: "v2022.5"
    
    versions:
        "v2022.5":
            prefix: "v2022.5/double-mpi"
            description:  Gromacs patched with Plumed v2.8.3
            dependencies:
                 - openmpi/4.1.5:intel-oneapi-2023
</file>

<file yaml gromacs-plumed.vpkg_yaml.caviness>
gromacs-plumed:
    prefix: /home/user/sw/Gromacs_Plumed/Gromacs
    description: Gromacs_Plumed
    url: "https://www.gromacs.org/"
    
    default-version: "v2022.5"
        
    versions:
        "v2022.5":
            prefix: "v2022.5/double-mpi"
            description:  Gromacs patched with Plumed v2.8.3
            dependencies:
                 - openmpi/4.1.4:intel-oneapi-2023
</file>

saved at ''$VALET_PKG_DIR/gromacs-plumed.vpkg_yaml''.

==== Checking the definition file ====

The package definition file can be checked for proper syntax using the VALET command ''vpkg_check'':
<code bash>
[user@login00.darwin ~]$ vpkg_check "$VALET_PKG_DIR/gromacs-plumed.vpkg_yaml"
/home/user/.valet/gromacs-plumed.vpkg_yaml is OK
[gromacs-plumed] {
  contexts: all
  actions: {
    GROMACS_PLUMED_PREFIX=${VALET_PATH_PREFIX} (contexts: development)
  }
  https://www.gromacs.org/
  Gromacs_Plumed
  prefix: /home/user/sw/Gromacs_Plumed/Gromacs
  source file: /home/user/.valet/gromacs-plumed.vpkg_yaml
  default version: gromacs-plumed/v2022.5
  versions: {
    [gromacs-plumed/v2022.5] {
      contexts: all
      dependencies: {
        openmpi/4.1.5:intel-oneapi-2023
      }
      Gromacs patched with Plumed v2.8.3
      prefix: /home/user/sw/Gromacs_Plumed/Gromacs/v2022.5/double-mpi
      standard paths: {
        bin: /home/user/sw/Gromacs_Plumed/Gromacs/v2022.5/double-mpi/bin
        lib: /home/user/sw/Gromacs_Plumed/Gromacs/v2022.5/double-mpi/lib64
        man: /home/user/sw/Gromacs_Plumed/Gromacs/v2022.5/double-mpi/share/man
        include: /home/user/sw/Gromacs_Plumed/Gromacs/v2022.5/double-mpi/include
      }
    }
  }
}

</code>
The file had no errors in its YAML syntax.  Notice also that the standard path (''bin'') is found and noted by VALET!

==== Runtime environment ====

To load **Gromacs_Plumed** v2022.5 into the runtime environment, the ''vpkg_require'' command is used:
<code bash>
[user@login00.darwin ~]$ vpkg_require gromacs-plumed/v2022.5
Adding dependency `binutils/2.35.1` to your environment
Adding dependency `gcc/12.2.0` to your environment
Adding dependency `intel-oneapi/2023.0.0.25537` to your environment
Adding dependency `ucx/1.13.1` to your environment
Adding dependency `openmpi/4.1.5:intel-oneapi-2023` to your environment
Adding package `gromacs-plumed/v2022.5` to your environment
[user@login00.darwin ~]$ which gmx_mpi_d
~/sw/Gromacs_Plumed/Gromacs/v2022.5/double-mpi/bin/gmx_mpi_d
</code>

===== Scaling =====

This is the wall time for MD run of 41057 atom, and nsteps=10000 with metadynamics by combining Gromacs and Plumed. The wall time decreases with an increased number of cores.

{{:technical:recipes:scale_gromacs_plumed.jpg?400|}}