hpc documentation - technical:recipes:vasp-wannier

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Building VASP with Wannier90 Support ======

The VASP electronic structure code can be extended with the Wannier90 library.  This document outlines a procedure for building VASP 6.3.2 + Wannier90 3.1.0 and integrating the product with the VALET package management system.

===== General Procedure =====

With an appropriate compiler toolchain loaded into the environment:

  - A package directory is created (if it does not yet exist) and a version directory is created under it
  - Wannier90 is built and installed to the version directory
  - VASP is built and linked against the installed Wannier90 library
  - The resulting VASP executables are copied to the version directory

Naturally, this procedure assumes the source packages for Wannier90 and VASP have already been downloaded.

The build procedure was performed on the DARWIN cluster.  Package names and versions as well as mechanisms for loading packages into the environment may differ between machines.

==== Load a Toolchain ====

VASP works well with the Intel compiler suite (Fortran and C) and MKL libraries for BLAS, LAPACK, ScaLAPACK, and FFT functions.  Coarse-grain parallelism is implemented with MPI, so an MPI compiler and runtime are also necessary.  On AMD systems the NVIDIA HPC SDK compiler suite (what used to be the Portland Compiler suite) can also be used.

Wannier90 must use the same Fortran and MPI compiler that will be used to build VASP.  It is also recommended to use the same BLAS/LAPACK library, e.g. MKL.

The MKL comes in two base variants:  sequential (serial) and threaded (OpenMP parallelism).  Using the threaded variant adds one level of hybrid parallelism to Wannier90 and VASP:  each MPI rank could use multiple threads (on distinct CPU cores) to parallelize BLAS/LAPACK operations.  Care must be exercised, though, since VASP itself includes optional OpenMP parallelism (enabled in the ''makefile.include'') which could interfere with MKL parallelism.  For this reason, we opt for the sequential MKL library and OpenMP parallelism in the VASP code in this example.

To that end, the toolchain is loaded into the environment:

<code bash>
$ vpkg_require openmpi/4.1.4:intel-2020
Adding dependency `intel/2020u4` to your environment
Adding dependency `ucx/1.13.1` to your environment
Adding package `openmpi/4.1.4:intel-2020` to your environment
</code>

==== Creating the Package Directory Hierarchy ====

On DARWIN every user has a home directory in which software can be compiled and stored.  For every workgroup to which the user is a member, there is workgroup storage that can also be used for that purpose:  if a software package is being maintained for the entire workgroup, it's best to store its variants there.  In this example it will be assumed that the software is maintained personally by and for the user.

A directory hierarchy is utilized to organize software packages and versions.  The user will be creating versions and variants of VASP, so the top-level directory will be:

<code bash>
$ VASP_PREFIX="${HOME}/sw/vasp"
$ mkdir -p "$VASP_PREFIX"
$ mkdir "${VASP_PREFIX}/attic"
</code>

The ''attic'' subdirectory is used to store the source code packages for VASP and any other components associated with that software -- so in this case, the Wannier90 source will be placed there, as well.  To download the Wannier90 3.1.0 source, for example:

<code bash>
$ pushd "${VASP_PREFIX}/attic"
$ wget "https://github.com/wannier-developers/wannier90/archive/v3.1.0.tar.gz"
$ popd
</code>

This recipe will be creating a variant of VASP 6.3.2 with Wannier90 3.1.0 functionality, so the version of the package will combine those attributes as well as the compiler toolchain used.  For the Intel compiler suite, then, the version will be ''6.3.2'' with attributes ''intel'' and ''wannier90-3.1.0'':

<code bash>
$ VASP_VERSION="6.3.2:wannier90-3.1.0,intel"
$ VASP_INSTALL_PATH="$(vpkg_id2path --package-prefix "$VASP_PREFIX" --version-id "$VASP_VERSION")"
$ echo "$VASP_INSTALL_PATH"
/home/1001/sw/vasp/6.3.2-intel-wannier90-3.1.0
$ mkdir "$VASP_INSTALL_PATH"
</code>

The source code will be colocated in that variant directory, so a ''src'' subdirectory will be created and the source packages unpacked therein:

<code bash>
$ mkdir "${VASP_INSTALL_PATH}/src"
$ cd "${VASP_INSTALL_PATH}/src"
$ tar -xf ../attic/v3.1.0.tar.gz
$ tar -xf ../attic/vasp.6.3.2.tar.gz
$ ls -l
total 36858
drwxr-xr-x  4 frey everyone        13 Feb  8 12:17 vasp.6.3.2
drwxr-xr-x 12 frey everyone        28 Feb  8 12:21 wannier90-3.1.0
</code>

==== Building Wannier90 ====

Next, Wannier90 is built.  The first step is to copy and modify a ''make.inc'' file for the build system:

<code bash>
$ cd wannier90-3.1.0
$ cp config/make.inc.ifort_tcm make.inc
</code>

The ''make.inc'' should be modified accordingly:

<file make make.inc>

#=====================================================
# For Linux with intel version 11/12 on 64bit machines
#=====================================================
F90 = ifort
COMMS=mpi
MPIF90=mpif90
FCOPTS=-O2
LDOPTS=-O2

#========================================================
# Intel mkl libraries. Set LIBPATH if not in default path
#========================================================

LIBDIR = 
LIBS   =  -mkl=sequential

#=======================
# ATLAS Blas and LAPACK
#=======================
#LIBDIR = /usr/local/lib
#LIBS = -L$(LIBDIR)  -llapack -lf77blas -lcblas -latlas
</file>

Note the use of the special Intel compiler flag ''-mkl'' which causes the Intel Fortran compiler to emit the necessary library paths and names to compile and link against the MKL library.  Were you building with threaded parallelism in the MKL library the flag would instead be ''-mkl=parallel''.

Build is now accomplished using ''make'' and the target ''all'':

<code bash>
$ make all
</code>

If successful, the completed library and executables can be installed in the VASP installation directory:

<code bash>
$ make PREFIX="$VASP_INSTALL_PATH" install
$ ls -l ../../lib
total 1506
-rw-r--r-- 1 frey everyone 3004552 Feb  8 11:52 libwannier.a

$ ls -l ../../bin
total 77578
-rwxr-xr-x 1 frey everyone  3119128 Feb  8 11:52 postw90.x
-rwxr-xr-x 1 frey everyone  3507440 Feb  8 11:52 w90chk2chk.x
-rwxr-xr-x 1 frey everyone  1074040 Feb  8 11:52 w90pov
-rwxr-xr-x 1 frey everyone  3483032 Feb  8 11:52 w90spn2spn.x
-rwxr-xr-x 1 frey everyone   988728 Feb  8 11:52 w90vdw.x
-rwxr-xr-x 1 frey everyone  3464584 Feb  8 11:52 wannier90.x
</code>

With the Wannier90 library completed, the VASP build can proceed.

==== Building VASP ====

The VASP source code is one directory level up; a customized copy of the Intel OpenMP + MKL + MPI architecture-specific file is required:

<code bash>
$ cd ../vasp.6.3.2
$ cp arch/makefile.include.intel_ompi_mkl_omp makefile.include
</code>

The following patch can be downloaded to the ''vasp.6.3.2'' source directory:

<file text makefile.include.patch>
--- A/makefile.include	2023-03-13 21:27:27.603020557 -0400
+++ B/makefile.include	2023-03-13 21:27:09.654568807 -0400
@@ -13,7 +13,7 @@
 CPP         = fpp -f_com=no -free -w0  $*$(FUFFIX) $*$(SUFFIX) $(CPP_OPTIONS)
 
 FC          = mpif90 -qopenmp
-FCL         = mpif90
+FCL         = mpif90 -qopenmp
 
 FREE        = -free -names lowercase
 
@@ -49,19 +49,19 @@
 
 # When compiling on the target machine itself, change this to the
 # relevant target when cross-compiling for another architecture
-VASP_TARGET_CPU ?= -xHOST
+VASP_TARGET_CPU ?= -xcore-avx2
 FFLAGS     += $(VASP_TARGET_CPU)
  
 # Intel MKL for FFTW, BLAS, LAPACK, and scaLAPACK
 # (Note: for Intel Parallel Studio's MKL use -mkl instead of -qmkl)
-FCL        += -qmkl
-MKLROOT    ?= /path/to/your/mkl/installation
+FCL        += -mkl=sequential
+MKLROOT    ?= 
 INCS        =-I$(MKLROOT)/include/fftw
 
 # Use a separate scaLAPACK installation (optional but recommended in combination with OpenMPI)
 # Comment out the two lines below if you want to use scaLAPACK from MKL instead
-SCALAPACK_ROOT ?= /path/to/your/scalapack/installation
-LLIBS      += -L${SCALAPACK_ROOT}/lib -lscalapack
+SCALAPACK_ROOT ?=
+LLIBS      += -L$(MKLROOT)/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64
 
 # HDF5-support (optional but strongly recommended)
 #CPP_OPTIONS+= -DVASP_HDF5
@@ -70,9 +70,9 @@
 #INCS       += -I$(HDF5_ROOT)/include
 
 # For the VASP-2-Wannier90 interface (optional)
-#CPP_OPTIONS    += -DVASP2WANNIER90
-#WANNIER90_ROOT ?= /path/to/your/wannier90/installation
-#LLIBS          += -L$(WANNIER90_ROOT)/lib -lwannier
+CPP_OPTIONS    += -DVASP2WANNIER90
+WANNIER90_ROOT ?=
+LLIBS          += -L$(WANNIER90_ROOT)/lib -lwannier
 
 # For the fftlib library (hardly any benefit in combination with MKL's FFTs)
 #CPP_OPTION += -Dsysv
</file>

and applied to the copy of the stock ''makefile.include'':

<code bash>
$ patch -p1 < makefile.include.patch
patching file makefile.include
</code>

In summary, the changes embodied in that patch are:

  * When the object code is linked using the ''$(FCL)'' command OpenMP libraries are only included if the ''-qopenmp'' flag is present
  * The AMD nodes in DARWIN have AVX2 capabilities but the ''-xHOST'' may not detect it; using ''-xcore-avx2'' forces AVX2 optimizations
  * The ''-qmkl'' flag isn't accepted by pre-2022 Intel suites and would default to the threaded library; use ''-mkl=sequential'' instead
  * ''$(MKLROOT)'' is present in the Intel suite runtime environment so no override value is necesssary
  * The Intel MKL ScaLAPACK and BLACS libraries are found under the ''$(MKLROOT)'' directory so a path relative to that is appropriate
  * The appropriate integer bit-depth libraries (lp64) with BLACS for Open MPI runtimes are needed
  * The Wannier90 library doesn't require a hard-coded path, we'll provide that from the environment; also, the Wannier90 flags should be uncommented

The build of VASP can now be effected:

<code bash>
$ make WANNIER90_ROOT="$VASP_INSTALL_PATH"
</code>

By default all three variants of VASP are produced -- std, gam, and ncl.  If the build is successful, the completed executables can be copied into the installation directory:

<code bash>
$ mkdir "${VASP_INSTALL_PATH}/bin"
$ cp bin/* "${VASP_INSTALL_PATH}/bin"
</code>

At this point all executables associated with Wannier90 and VASP are present in the ''${VASP_INSTALL_PATH}/bin'' directory and the Wannier90 library is present in ''${VASP_INSTALL_PATH}/lib''.  To make use of this variant of VASP, the ''${VASP_INSTALL_PATH}/bin'' directory must be added to the ''PATH'' environment variable.  That is where VALET comes in.

==== VALET Package Definition ====

Since this copy of VASP resides in the user's home directory, the appropriate location to install a VALET package definition for it is ''~/.valet''.  This directory is present by default for all DARWIN users.

Key values to know before editing the package definition were assigned to variables throughout this recipe:

<code bash>
$ for V in VASP_VERSION VASP_PREFIX; do echo "$V = $(eval echo \$$V)"; done
VASP_VERSION = 6.3.2:wannier90-3.1.0,intel
VASP_PREFIX = /home/1001/sw/vasp

$ vpkg_history
[standard]
  intel/2020u4
  ucx/1.13.1
  openmpi/4.1.4:intel-2020
</code>

Recall that we added ''openmpi/4.1.4:intel-2020'' to the environment with ''vpkg_require''; that is the single direct dependency for this variant of VASP.  The following template can be saved to ''~/.valet/vasp.vpkg_yaml'' and edited to include the values of those variables:

<file yaml vasp.vpkg_yaml>
vasp:
    prefix: $VASP_PREFIX
    description: Vienna Ab initio Simulation Package
    url: http://www.vasp.at
    
    default-version: $VASP_VERSION
    versions:
        "$VASP_VERSION":
            description: VASP 6.3.2, Wannier90 3.1.0, Intel compilers and MKL
            dependencies:
                - openmpi/4.1.4:intel-2020
</file>

VALET can now be used to query versions of VASP available:

<code bash>
$ vpkg_versions vasp

Available versions/variants in package (* = default version):

[/home/1001/.valet/vasp.vpkg_yaml]
vasp                           Vienna Ab initio Simulation Package
* 6.3.2:intel,wannier90-3.1.0  VASP 6.3.2, Wannier90 3.1.0, Intel compilers and MKL
</code>

===== Using VASP =====

Henceforth, this variant of VASP 6.3.2 can be added to your shell environment using VALET:

<code bash>
$ vpkg_require vasp/6.3.2:wannier90-3.1.0,intel
Adding dependency `intel/2020u4` to your environment
Adding dependency `ucx/1.13.1` to your environment
Adding dependency `openmpi/4.1.4:intel-2020` to your environment
Adding package `vasp/6.3.2:wannier90-3.1.0,intel` to your environment

$ which vasp_std
/home/1001/sw/vasp/6.3.2-intel-wannier90-3.1.0/bin/vasp_std

$ which wannier90.x
/home/1001/sw/vasp/6.3.2-intel-wannier90-3.1.0/bin/wannier90.x
</code>

The same ''vpkg_require'' command can be used in a job script to setup the runtime environment for the execution of VASP.  The Open MPI job script template at ''/opt/templates/slurm/generic/mpi/openmpi/openmpi.qs'' can be copied and modified accordingly:

<code text>
   :
#
# [EDIT] Do any pre-processing, staging, environment setup with VALET
#        or explicit changes to PATH, LD_LIBRARY_PATH, etc.
#
vpkg_require vasp/6.3.2:wannier90-3.1.0,intel
   :
#
# [EDIT] Execute your MPI program
#
${UD_MPIRUN} vasp_std 
mpi_rc=$?

#
# [EDIT] Do any cleanup work here...
#

#
# Be sure to return the mpirun's result code:
#
exit $mpi_rc
</code>