====== Building VASP with Wannier90 Support ====== The VASP electronic structure code can be extended with the Wannier90 library. This document outlines a procedure for building VASP 6.3.2 + Wannier90 3.1.0 and integrating the product with the VALET package management system. ===== General Procedure ===== With an appropriate compiler toolchain loaded into the environment: - A package directory is created (if it does not yet exist) and a version directory is created under it - Wannier90 is built and installed to the version directory - VASP is built and linked against the installed Wannier90 library - The resulting VASP executables are copied to the version directory Naturally, this procedure assumes the source packages for Wannier90 and VASP have already been downloaded. The build procedure was performed on the DARWIN cluster. Package names and versions as well as mechanisms for loading packages into the environment may differ between machines. ==== Load a Toolchain ==== VASP works well with the Intel compiler suite (Fortran and C) and MKL libraries for BLAS, LAPACK, ScaLAPACK, and FFT functions. Coarse-grain parallelism is implemented with MPI, so an MPI compiler and runtime are also necessary. On AMD systems the NVIDIA HPC SDK compiler suite (what used to be the Portland Compiler suite) can also be used. Wannier90 must use the same Fortran and MPI compiler that will be used to build VASP. It is also recommended to use the same BLAS/LAPACK library, e.g. MKL. The MKL comes in two base variants: sequential (serial) and threaded (OpenMP parallelism). Using the threaded variant adds one level of hybrid parallelism to Wannier90 and VASP: each MPI rank could use multiple threads (on distinct CPU cores) to parallelize BLAS/LAPACK operations. Care must be exercised, though, since VASP itself includes optional OpenMP parallelism (enabled in the ''makefile.include'') which could interfere with MKL parallelism. For this reason, we opt for the sequential MKL library and OpenMP parallelism in the VASP code in this example. To that end, the toolchain is loaded into the environment: $ vpkg_require openmpi/4.1.4:intel-2020 Adding dependency `intel/2020u4` to your environment Adding dependency `ucx/1.13.1` to your environment Adding package `openmpi/4.1.4:intel-2020` to your environment ==== Creating the Package Directory Hierarchy ==== On DARWIN every user has a home directory in which software can be compiled and stored. For every workgroup to which the user is a member, there is workgroup storage that can also be used for that purpose: if a software package is being maintained for the entire workgroup, it's best to store its variants there. In this example it will be assumed that the software is maintained personally by and for the user. A directory hierarchy is utilized to organize software packages and versions. The user will be creating versions and variants of VASP, so the top-level directory will be: $ VASP_PREFIX="${HOME}/sw/vasp" $ mkdir -p "$VASP_PREFIX" $ mkdir "${VASP_PREFIX}/attic" The ''attic'' subdirectory is used to store the source code packages for VASP and any other components associated with that software -- so in this case, the Wannier90 source will be placed there, as well. To download the Wannier90 3.1.0 source, for example: $ pushd "${VASP_PREFIX}/attic" $ wget "https://github.com/wannier-developers/wannier90/archive/v3.1.0.tar.gz" $ popd This recipe will be creating a variant of VASP 6.3.2 with Wannier90 3.1.0 functionality, so the version of the package will combine those attributes as well as the compiler toolchain used. For the Intel compiler suite, then, the version will be ''6.3.2'' with attributes ''intel'' and ''wannier90-3.1.0'': $ VASP_VERSION="6.3.2:wannier90-3.1.0,intel" $ VASP_INSTALL_PATH="$(vpkg_id2path --package-prefix "$VASP_PREFIX" --version-id "$VASP_VERSION")" $ echo "$VASP_INSTALL_PATH" /home/1001/sw/vasp/6.3.2-intel-wannier90-3.1.0 $ mkdir "$VASP_INSTALL_PATH" The source code will be colocated in that variant directory, so a ''src'' subdirectory will be created and the source packages unpacked therein: $ mkdir "${VASP_INSTALL_PATH}/src" $ cd "${VASP_INSTALL_PATH}/src" $ tar -xf ../attic/v3.1.0.tar.gz $ tar -xf ../attic/vasp.6.3.2.tar.gz $ ls -l total 36858 drwxr-xr-x 4 frey everyone 13 Feb 8 12:17 vasp.6.3.2 drwxr-xr-x 12 frey everyone 28 Feb 8 12:21 wannier90-3.1.0 ==== Building Wannier90 ==== Next, Wannier90 is built. The first step is to copy and modify a ''make.inc'' file for the build system: $ cd wannier90-3.1.0 $ cp config/make.inc.ifort_tcm make.inc The ''make.inc'' should be modified accordingly: #===================================================== # For Linux with intel version 11/12 on 64bit machines #===================================================== F90 = ifort COMMS=mpi MPIF90=mpif90 FCOPTS=-O2 LDOPTS=-O2 #======================================================== # Intel mkl libraries. Set LIBPATH if not in default path #======================================================== LIBDIR = LIBS = -mkl=sequential #======================= # ATLAS Blas and LAPACK #======================= #LIBDIR = /usr/local/lib #LIBS = -L$(LIBDIR) -llapack -lf77blas -lcblas -latlas Note the use of the special Intel compiler flag ''-mkl'' which causes the Intel Fortran compiler to emit the necessary library paths and names to compile and link against the MKL library. Were you building with threaded parallelism in the MKL library the flag would instead be ''-mkl=parallel''. Build is now accomplished using ''make'' and the target ''all'': $ make all If successful, the completed library and executables can be installed in the VASP installation directory: $ make PREFIX="$VASP_INSTALL_PATH" install $ ls -l ../../lib total 1506 -rw-r--r-- 1 frey everyone 3004552 Feb 8 11:52 libwannier.a $ ls -l ../../bin total 77578 -rwxr-xr-x 1 frey everyone 3119128 Feb 8 11:52 postw90.x -rwxr-xr-x 1 frey everyone 3507440 Feb 8 11:52 w90chk2chk.x -rwxr-xr-x 1 frey everyone 1074040 Feb 8 11:52 w90pov -rwxr-xr-x 1 frey everyone 3483032 Feb 8 11:52 w90spn2spn.x -rwxr-xr-x 1 frey everyone 988728 Feb 8 11:52 w90vdw.x -rwxr-xr-x 1 frey everyone 3464584 Feb 8 11:52 wannier90.x With the Wannier90 library completed, the VASP build can proceed. ==== Building VASP ==== The VASP source code is one directory level up; a customized copy of the Intel OpenMP + MKL + MPI architecture-specific file is required: $ cd ../vasp.6.3.2 $ cp arch/makefile.include.intel_ompi_mkl_omp makefile.include The following patch can be downloaded to the ''vasp.6.3.2'' source directory: --- A/makefile.include 2023-03-13 21:27:27.603020557 -0400 +++ B/makefile.include 2023-03-13 21:27:09.654568807 -0400 @@ -13,7 +13,7 @@ CPP = fpp -f_com=no -free -w0 $*$(FUFFIX) $*$(SUFFIX) $(CPP_OPTIONS) FC = mpif90 -qopenmp -FCL = mpif90 +FCL = mpif90 -qopenmp FREE = -free -names lowercase @@ -49,19 +49,19 @@ # When compiling on the target machine itself, change this to the # relevant target when cross-compiling for another architecture -VASP_TARGET_CPU ?= -xHOST +VASP_TARGET_CPU ?= -xcore-avx2 FFLAGS += $(VASP_TARGET_CPU) # Intel MKL for FFTW, BLAS, LAPACK, and scaLAPACK # (Note: for Intel Parallel Studio's MKL use -mkl instead of -qmkl) -FCL += -qmkl -MKLROOT ?= /path/to/your/mkl/installation +FCL += -mkl=sequential +MKLROOT ?= INCS =-I$(MKLROOT)/include/fftw # Use a separate scaLAPACK installation (optional but recommended in combination with OpenMPI) # Comment out the two lines below if you want to use scaLAPACK from MKL instead -SCALAPACK_ROOT ?= /path/to/your/scalapack/installation -LLIBS += -L${SCALAPACK_ROOT}/lib -lscalapack +SCALAPACK_ROOT ?= +LLIBS += -L$(MKLROOT)/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 # HDF5-support (optional but strongly recommended) #CPP_OPTIONS+= -DVASP_HDF5 @@ -70,9 +70,9 @@ #INCS += -I$(HDF5_ROOT)/include # For the VASP-2-Wannier90 interface (optional) -#CPP_OPTIONS += -DVASP2WANNIER90 -#WANNIER90_ROOT ?= /path/to/your/wannier90/installation -#LLIBS += -L$(WANNIER90_ROOT)/lib -lwannier +CPP_OPTIONS += -DVASP2WANNIER90 +WANNIER90_ROOT ?= +LLIBS += -L$(WANNIER90_ROOT)/lib -lwannier # For the fftlib library (hardly any benefit in combination with MKL's FFTs) #CPP_OPTION += -Dsysv and applied to the copy of the stock ''makefile.include'': $ patch -p1 < makefile.include.patch patching file makefile.include In summary, the changes embodied in that patch are: * When the object code is linked using the ''$(FCL)'' command OpenMP libraries are only included if the ''-qopenmp'' flag is present * The AMD nodes in DARWIN have AVX2 capabilities but the ''-xHOST'' may not detect it; using ''-xcore-avx2'' forces AVX2 optimizations * The ''-qmkl'' flag isn't accepted by pre-2022 Intel suites and would default to the threaded library; use ''-mkl=sequential'' instead * ''$(MKLROOT)'' is present in the Intel suite runtime environment so no override value is necesssary * The Intel MKL ScaLAPACK and BLACS libraries are found under the ''$(MKLROOT)'' directory so a path relative to that is appropriate * The appropriate integer bit-depth libraries (lp64) with BLACS for Open MPI runtimes are needed * The Wannier90 library doesn't require a hard-coded path, we'll provide that from the environment; also, the Wannier90 flags should be uncommented The build of VASP can now be effected: $ make WANNIER90_ROOT="$VASP_INSTALL_PATH" By default all three variants of VASP are produced -- std, gam, and ncl. If the build is successful, the completed executables can be copied into the installation directory: $ mkdir "${VASP_INSTALL_PATH}/bin" $ cp bin/* "${VASP_INSTALL_PATH}/bin" At this point all executables associated with Wannier90 and VASP are present in the ''${VASP_INSTALL_PATH}/bin'' directory and the Wannier90 library is present in ''${VASP_INSTALL_PATH}/lib''. To make use of this variant of VASP, the ''${VASP_INSTALL_PATH}/bin'' directory must be added to the ''PATH'' environment variable. That is where VALET comes in. ==== VALET Package Definition ==== Since this copy of VASP resides in the user's home directory, the appropriate location to install a VALET package definition for it is ''~/.valet''. This directory is present by default for all DARWIN users. Key values to know before editing the package definition were assigned to variables throughout this recipe: $ for V in VASP_VERSION VASP_PREFIX; do echo "$V = $(eval echo \$$V)"; done VASP_VERSION = 6.3.2:wannier90-3.1.0,intel VASP_PREFIX = /home/1001/sw/vasp $ vpkg_history [standard] intel/2020u4 ucx/1.13.1 openmpi/4.1.4:intel-2020 Recall that we added ''openmpi/4.1.4:intel-2020'' to the environment with ''vpkg_require''; that is the single direct dependency for this variant of VASP. The following template can be saved to ''~/.valet/vasp.vpkg_yaml'' and edited to include the values of those variables: vasp: prefix: $VASP_PREFIX description: Vienna Ab initio Simulation Package url: http://www.vasp.at default-version: $VASP_VERSION versions: "$VASP_VERSION": description: VASP 6.3.2, Wannier90 3.1.0, Intel compilers and MKL dependencies: - openmpi/4.1.4:intel-2020 VALET can now be used to query versions of VASP available: $ vpkg_versions vasp Available versions/variants in package (* = default version): [/home/1001/.valet/vasp.vpkg_yaml] vasp Vienna Ab initio Simulation Package * 6.3.2:intel,wannier90-3.1.0 VASP 6.3.2, Wannier90 3.1.0, Intel compilers and MKL ===== Using VASP ===== Henceforth, this variant of VASP 6.3.2 can be added to your shell environment using VALET: $ vpkg_require vasp/6.3.2:wannier90-3.1.0,intel Adding dependency `intel/2020u4` to your environment Adding dependency `ucx/1.13.1` to your environment Adding dependency `openmpi/4.1.4:intel-2020` to your environment Adding package `vasp/6.3.2:wannier90-3.1.0,intel` to your environment $ which vasp_std /home/1001/sw/vasp/6.3.2-intel-wannier90-3.1.0/bin/vasp_std $ which wannier90.x /home/1001/sw/vasp/6.3.2-intel-wannier90-3.1.0/bin/wannier90.x The same ''vpkg_require'' command can be used in a job script to setup the runtime environment for the execution of VASP. The Open MPI job script template at ''/opt/templates/slurm/generic/mpi/openmpi/openmpi.qs'' can be copied and modified accordingly: : # # [EDIT] Do any pre-processing, staging, environment setup with VALET # or explicit changes to PATH, LD_LIBRARY_PATH, etc. # vpkg_require vasp/6.3.2:wannier90-3.1.0,intel : # # [EDIT] Execute your MPI program # ${UD_MPIRUN} vasp_std mpi_rc=$? # # [EDIT] Do any cleanup work here... # # # Be sure to return the mpirun's result code: # exit $mpi_rc