Building Gromacs-Plumed on Caviness/DARWIN

Building Gromacs-Plumed on Caviness/DARWIN

The build procedure outlined herein uses Open MPI on top of the Intel compiler suite.

Directory Preparation

To begin, choose a directory in which the Gromacs_Plumed version(s) will be built and installed. To build in your home directory, for example:

[user@login00.darwin ~]$ GP_BASEDIR=~/sw/Gromacs_Plumed
[user@login00.darwin ~]$ G_BASEDIR="${GP_BASEDIR}/Gromacs"
[user@login00.darwin ~]$ P_BASEDIR="${GP_BASEDIR}/Plumed"
[user@login00.darwin ~]$ GP_BASEDIR_PRIVS=0700

If you are managing Gromacs_Plumed software for your entire workgroup, you could instead use

[user@login00.darwin ~]$ GP_BASEDIR="${WORKDIR}/sw/Gromacs_Plumed"
[user@login00.darwin ~]$ G_BASEDIR="${GP_BASEDIR}/Gromacs"
[user@login00.darwin ~]$ P_BASEDIR="${GP_BASEDIR}/Plumed"
[user@login00.darwin ~]$ GP_BASEDIR_PRIVS=2770

If the directory hierarchy does not yet exist, it can be setup using

[user@login00.darwin ~]$ mkdir -p -m $GP_BASEDIR_PRIVS "${GP_BASEDIR}"
[user@login00.darwin ~]$ mkdir -p -m $GP_BASEDIR_PRIVS "${G_BASEDIR}"
[user@login00.darwin ~]$ mkdir -p -m $GP_BASEDIR_PRIVS "${P_BASEDIR}"

Source Preparation: Plumed

In this example version 2.8.3 of Plumed will be built. Our standard recipes for Plumed will entail use of the Intel compilers, the MKL for BLAS/LAPACK/FFTW/ScaLAPACK/BLACS, and Open MPI for parallelism.

Download and unpack "plumed-2.8.3.tgz" and create a directory to hold our base build of plumed-2.8.3, naming it with the version identifier: v2.8.3. The source is then unpacked therein:

[user@login00.darwin ~]$ cd $P_BASEDIR
[user@login00.darwin Plumed]$ wget https://github.com/plumed/plumed2/releases/download/v2.8.3/plumed-2.8.3.tgz
[user@login00.darwin Plumed]$ tar -xvzf plumed-2.8.3.tgz

Compilation and Installation: Plumed

We use configure for compilation and installation on "v2.8.3" directory.

[user@login00.darwin ~]$ P_INSTALL_PREFIX="${P_BASEDIR}/v2.8.3"
[user@login00.darwin ~]$ export P_INSTALL_PREFIX
[user@login00.darwin ~]$ P_SRC_PREFIX="${P_BASEDIR}/plumed-2.8.3"
[user@login00.darwin ~]$ mkdir -m $GP_BASEDIR_PRIVS "$P_INSTALL_PREFIX"
[user@login00.darwin ~]$ cd "$P_SRC_PREFIX"
[user@login00.darwin plumed-2.8.3]$ vpkg_devrequire openmpi/4.1.5:intel-oneapi-2023
[user@login00.darwin plumed-2.8.3]$ ./configure --prefix=$P_INSTALL_PREFIX  CC=mpicc CXX=mpicxx FC=mpifort  LDFLAGS=-qmkl=sequential 
[user@login00.darwin plumed-2.8.3]$ make
[user@login00.darwin plumed-2.8.3]$ make install

Our current working directory is now the source directory. Here, we create sourceme.sh :

sourceme.sh

TPATH="${P_INSTALL_PREFIX}"
export PATH="$TPATH/bin/:$PATH"
export LIBRARY_PATH="$TPATH/lib/:$LIBRARY_PATH"
export LD_LIBRARY_PATH="$TPATH/lib/:$LD_LIBRARY_PATH"
export PLUMED_KERNEL="$TPATH/lib/libplumedKernel.so"
export PLUMED_VIMPATH="$TPATH/lib/plumed/vim/"
export PYTHONPATH="$TPATH/lib/python/:$PYTHONPATH"
 
export CPATH="$TPATH/include/:$CPATH”
export INCLUDE="$TPATH/include/:$INCLUDE”

Executing the script.

[user@login00.darwin plumed-2.8.3]$ source sourceme.sh

Source Preparation: Gromacs

In this example version 2022.5 of Gromacs will be built. Our standard recipes for Gromacs will entail use of the Intel compilers, the MKL for BLAS/LAPACK/FFTW/ScaLAPACK/BLACS, and Open MPI for parallelism.

Download and unpack "gromacs-2022.5.tar.gz" and create a directory to hold our base build of gromacs-2022.5, naming it with the version identifier: v2022.5. The source is then unpacked therein:

[user@login00.darwin plumed-2.8.3]$ cd $G_BASEDIR
[user@login00.darwin Gromacs]$ wget https://ftp.gromacs.org/gromacs/gromacs-2022.5.tar.gz
[user@login00.darwin Gromacs]$ tar -xvzf gromacs-2022.5.tar.gz

Compilation and Installation: Plumed-patched Gromacs

We use cmake for compilation and installation on Plumed-patched Gromacs "v2022.5" directory.

[user@login00.darwin Gromacs]$ G_INSTALL_PREFIX="${G_BASEDIR}/v2022.5"
[user@login00.darwin Gromacs]$ export G_INSTALL_PREFIX
[user@login00.darwin Gromacs]$ G_SRC_PREFIX="${G_BASEDIR}/gromacs-2022.5"
[user@login00.darwin Gromacs]$ mkdir -m $GP_BASEDIR_PRIVS "$G_INSTALL_PREFIX"
[user@login00.darwin Gromacs]$ G_BUILDDIR="${G_SRC_PREFIX}/build"
[user@login00.darwin Gromacs]$ mkdir -m $GP_BASEDIR_PRIVS "$G_BUILDDIR"
[user@login00.darwin Gromacs]$ cd "$G_SRC_PREFIX"

Hear, we patch Plumed on Gromacs.

[user@login00.darwin gromacs-2022.5]$ plumed patch -p --static

Select 4:

1) gromacs-2019.6     5) gromacs-4.5.7	   9) qespresso-5.0.2
2) gromacs-2020.7     6) namd-2.12	  10) qespresso-6.2
3) gromacs-2021.7     7) namd-2.13
4) gromacs-2022.5     8) namd-2.14

[user@login00.darwin gromacs-2022.5]$ cd "$G_BUILDDIR"

Our current working directory is now the build root. Here, we create SWMGR-build.sh :

SWMGR-build.sh.darwin

#!/bin/bash -l
 
vpkg_require cmake/default openmpi/4.1.5:intel-oneapi-2023
 
PREFIX="${G_INSTALL_PREFIX}"
 
CPU_GMX_SIMD=AVX2_256
 
BASE_BUILD_FLAGS="
    -DCMAKE_BUILD_TYPE=Release
    -DGMX_SIMD=${CPU_GMX_SIMD}
    -DGMX_FFT_LIBRARY=mkl
    -DMKL_MPI=openmpi
    -DBUILD_SHARED_LIBS=OFF
    -DGMX_PREFER_STATIC_LIBS=ON
"
# BASE_BUILD_FLAGS="$BASE_BUILD_FLAGS -DGMX_GPU=SYCL"
#    -DMKL_INCLUDE_DIR=$MKLROOT/include
#    -DMKL_LIBRARIES=-mkl=sequential
#"
 
build_variant()
{
    local DBL=$1 MPI=$2
    local build_dir=""
    local build_flags=""
 
    if [ $DBL -ne 0 ]; then
        build_dir="double"
	build_flags="${build_flags} -DGMX_DOUBLE=on"
    else
        build_dir="single"
        build_flags="${build_flags} -DGMX_DOUBLE=off"
    fi
    if [ $MPI -eq 0 ]; then
        build_dir="${build_dir}-thread"
        build_flags="${build_flags} -DGMX_MPI=off -DGMX_THREAD_MPI=on -DCMAKE_DISABLE_FIND_PACKAGE_MPI=on"
        CC=icx CXX=icpx FC=ifx
    else
        build_dir="${build_dir}-mpi"
        build_flags="${build_flags} -DGMX_MPI=on -DGMX_THREAD_MPI=off"
	CC=mpicc CXX=mpicxx FC=mpifort
    fi
    [ -d "build-${build_dir}" ] && rm -rf "build-${build_dir}"
    mkdir "build-${build_dir}" ; pushd "build-${build_dir}"
    CC=$CC CXX=$CXX FC=$FC \
        cmake \
	    -DCMAKE_INSTALL_PREFIX="${PREFIX}/${build_dir}" \
            $BASE_BUILD_FLAGS $build_flags \
	../..
    if [ $? -eq 0 ]; then
        make -j 20
        if [ $? -eq 0 ]; then
            make install
        fi
    fi
    popd
}
 
##
## SYCL only supports single precision.
##
#             DOUBLE?    MPI?
#build_variant 0          0
#build_variant 1          0
#build_variant 0          1
build_variant 1          1

SWMGR-build.sh.caviness

#!/bin/bash -l
 
vpkg_require cmake/default openmpi/4.1.4:intel-oneapi-2023
 
PREFIX="${G_INSTALL_PREFIX}"
 
CPU_GMX_SIMD=AVX2_256
 
BASE_BUILD_FLAGS="
    -DCMAKE_BUILD_TYPE=Release
    -DGMX_SIMD=${CPU_GMX_SIMD}
    -DGMX_FFT_LIBRARY=mkl
    -DMKL_MPI=openmpi
    -DBUILD_SHARED_LIBS=OFF
    -DGMX_PREFER_STATIC_LIBS=ON
"
# BASE_BUILD_FLAGS="$BASE_BUILD_FLAGS -DGMX_GPU=SYCL"
#    -DMKL_INCLUDE_DIR=$MKLROOT/include
#    -DMKL_LIBRARIES=-mkl=sequential
#"
 
build_variant()
{
    local DBL=$1 MPI=$2
    local build_dir=""
    local build_flags=""
 
    if [ $DBL -ne 0 ]; then
        build_dir="double"
	build_flags="${build_flags} -DGMX_DOUBLE=on"
    else
        build_dir="single"
        build_flags="${build_flags} -DGMX_DOUBLE=off"
    fi
    if [ $MPI -eq 0 ]; then
        build_dir="${build_dir}-thread"
        build_flags="${build_flags} -DGMX_MPI=off -DGMX_THREAD_MPI=on -DCMAKE_DISABLE_FIND_PACKAGE_MPI=on"
        CC=icx CXX=icpx FC=ifx
    else
        build_dir="${build_dir}-mpi"
        build_flags="${build_flags} -DGMX_MPI=on -DGMX_THREAD_MPI=off"
	CC=mpicc CXX=mpicxx FC=mpifort
    fi
    [ -d "build-${build_dir}" ] && rm -rf "build-${build_dir}"
    mkdir "build-${build_dir}" ; pushd "build-${build_dir}"
    CC=$CC CXX=$CXX FC=$FC \
        cmake \
	    -DCMAKE_INSTALL_PREFIX="${PREFIX}/${build_dir}" \
            $BASE_BUILD_FLAGS $build_flags \
	../..
    if [ $? -eq 0 ]; then
        make -j 20
        if [ $? -eq 0 ]; then
            make install
        fi
    fi
    popd
}
 
##
## SYCL only supports single precision.
##
#             DOUBLE?    MPI?
#build_variant 0          0
#build_variant 1          0
#build_variant 0          1
build_variant 1          1

Executing the script compile and install plumed-patched gromacs.

[user@login00.darwin build]$ ./SWMGR-build.sh

VALET Package Definition

With this version of Gromacs_Plumed built, the remaining step is to leverage VALET for setup of the runtime environment when you use the software. VALET automatically recognizes the standard directory layout, so configuring versions/variants of Gromacs_Plumed is very straightforward. First, note your installation path:

[user@login00.darwin build]$ vpkg_rollback all
[user@login00.darwin build]$ cd
[user@login00.darwin ~]$ echo $GP_BASEDIR
/home/user/sw/Gromacs_Plumed

Since this build was done in the user's home directory, they were personal copies of the software and should use a VALET package definition file stored in ~/.valet

[user@login00.darwin ~]$ VALET_PKG_DIR=~/.valet ; VALET_PKG_DIR_MODE=0700

versus an installation made for an entire workgroup, which would store the VALET package definition files in $WORKDIR/sw/valet

[user@login00.darwin ~]$ VALET_PKG_DIR="$WORKDIR/sw/valet" ; VALET_PKG_DIR_MODE=2770

Whichever scheme is in-use, ensure the directory exists:

[user@login00.darwin ~]$ mkdir -p --mode=$VALET_PKG_DIR_MODE "$VALET_PKG_DIR"

VALET allows package definitions in a variety of formats (XML, JSON, YAML) but YAML tends to be the simplest format so we will use it here.

Package section

The package section of the definition file includes items that apply to all versions/variants of the software:

Gromacs_Plumed:
    prefix: /home/user/sw/Gromacs_Plumed/Gromacs
    description: Gromacs_Plumed
    url: "https://www.gromacs.org/"

The package identifier is the top-level key in the document — Gromacs_Plumed — and the value of $GP_BASEDIR is the value of the prefix key in this section. The URL and description are information taken from the official Gromacs web site.

Versions

The versions key is used to provide a list of the versions/variants of the software:

gromacs-plumed:
    prefix: /home/user/sw/Gromacs_Plumed/Gromacs
    description: Gromacs_Plumed
    url: "https://www.gromacs.org/"
    
    default-version: "v2022.5"
    
    versions:
        "v2022.5":
            prefix: "v2022.5/double-mpi"
            description:  Gromacs patched with Plumed v2.8.3
            dependencies:
                 - openmpi/4.1.5:intel-oneapi-2023

The version identifier v7.3 is inferred to be the path prefix to the version in question here. The package's prefix (/home/user/sw/Gromacs_Plumed) with the version identifier appended (/home/user/sw/Gromacs_Plumed/Gromacs/v2022.5) is implicit.

The implicit behavior is overridden by providing a prefix key in the version definition: a relative path is appended to the package's prefix, an absolute path is used as-is.

It is a good idea to specify which version definition should act as the default. This yields the following package definition file

gromacs-plumed.vpkg_yaml.darwin

gromacs-plumed:
    prefix: /home/user/sw/Gromacs_Plumed/Gromacs
    description: Gromacs_Plumed
    url: "https://www.gromacs.org/"
    
    default-version: "v2022.5"
    
    versions:
        "v2022.5":
            prefix: "v2022.5/double-mpi"
            description:  Gromacs patched with Plumed v2.8.3
            dependencies:
                 - openmpi/4.1.5:intel-oneapi-2023

gromacs-plumed.vpkg_yaml.caviness

gromacs-plumed:
    prefix: /home/user/sw/Gromacs_Plumed/Gromacs
    description: Gromacs_Plumed
    url: "https://www.gromacs.org/"
    
    default-version: "v2022.5"
        
    versions:
        "v2022.5":
            prefix: "v2022.5/double-mpi"
            description:  Gromacs patched with Plumed v2.8.3
            dependencies:
                 - openmpi/4.1.4:intel-oneapi-2023

saved at $VALET_PKG_DIR/gromacs-plumed.vpkg_yaml.

Checking the definition file

The package definition file can be checked for proper syntax using the VALET command vpkg_check:

[user@login00.darwin ~]$ vpkg_check "$VALET_PKG_DIR/gromacs-plumed.vpkg_yaml"
/home/user/.valet/gromacs-plumed.vpkg_yaml is OK
[gromacs-plumed] {
  contexts: all
  actions: {
    GROMACS_PLUMED_PREFIX=${VALET_PATH_PREFIX} (contexts: development)
  }
  https://www.gromacs.org/
  Gromacs_Plumed
  prefix: /home/user/sw/Gromacs_Plumed/Gromacs
  source file: /home/user/.valet/gromacs-plumed.vpkg_yaml
  default version: gromacs-plumed/v2022.5
  versions: {
    [gromacs-plumed/v2022.5] {
      contexts: all
      dependencies: {
        openmpi/4.1.5:intel-oneapi-2023
      }
      Gromacs patched with Plumed v2.8.3
      prefix: /home/user/sw/Gromacs_Plumed/Gromacs/v2022.5/double-mpi
      standard paths: {
        bin: /home/user/sw/Gromacs_Plumed/Gromacs/v2022.5/double-mpi/bin
        lib: /home/user/sw/Gromacs_Plumed/Gromacs/v2022.5/double-mpi/lib64
        man: /home/user/sw/Gromacs_Plumed/Gromacs/v2022.5/double-mpi/share/man
        include: /home/user/sw/Gromacs_Plumed/Gromacs/v2022.5/double-mpi/include
      }
    }
  }
}

The file had no errors in its YAML syntax. Notice also that the standard path (bin) is found and noted by VALET!

Runtime environment

To load Gromacs_Plumed v2022.5 into the runtime environment, the vpkg_require command is used:

[user@login00.darwin ~]$ vpkg_require gromacs-plumed/v2022.5
Adding dependency `binutils/2.35.1` to your environment
Adding dependency `gcc/12.2.0` to your environment
Adding dependency `intel-oneapi/2023.0.0.25537` to your environment
Adding dependency `ucx/1.13.1` to your environment
Adding dependency `openmpi/4.1.5:intel-oneapi-2023` to your environment
Adding package `gromacs-plumed/v2022.5` to your environment
[user@login00.darwin ~]$ which gmx_mpi_d
~/sw/Gromacs_Plumed/Gromacs/v2022.5/double-mpi/bin/gmx_mpi_d

Scaling

This is the wall time for MD run of 41057 atom, and nsteps=10000 with metadynamics by combining Gromacs and Plumed. The wall time decreases with an increased number of cores.

Table of Contents