technical:recipes:vasp-wannier

Building VASP with Wannier90 Support

The VASP electronic structure code can be extended with the Wannier90 library. This document outlines a procedure for building VASP 6.3.2 + Wannier90 3.1.0 and integrating the product with the VALET package management system.

General Procedure

With an appropriate compiler toolchain loaded into the environment:

  1. A package directory is created (if it does not yet exist) and a version directory is created under it
  2. Wannier90 is built and installed to the version directory
  3. VASP is built and linked against the installed Wannier90 library
  4. The resulting VASP executables are copied to the version directory

Naturally, this procedure assumes the source packages for Wannier90 and VASP have already been downloaded.

The build procedure was performed on the DARWIN cluster. Package names and versions as well as mechanisms for loading packages into the environment may differ between machines.

Load a Toolchain

VASP works well with the Intel compiler suite (Fortran and C) and MKL libraries for BLAS, LAPACK, ScaLAPACK, and FFT functions. Coarse-grain parallelism is implemented with MPI, so an MPI compiler and runtime are also necessary. On AMD systems the NVIDIA HPC SDK compiler suite (what used to be the Portland Compiler suite) can also be used.

Wannier90 must use the same Fortran and MPI compiler that will be used to build VASP. It is also recommended to use the same BLAS/LAPACK library, e.g. MKL.

The MKL comes in two base variants: sequential (serial) and threaded (OpenMP parallelism). Using the threaded variant adds one level of hybrid parallelism to Wannier90 and VASP: each MPI rank could use multiple threads (on distinct CPU cores) to parallelize BLAS/LAPACK operations. Care must be exercised, though, since VASP itself includes optional OpenMP parallelism (enabled in the makefile.include) which could interfere with MKL parallelism. For this reason, we opt for the sequential MKL library and OpenMP parallelism in the VASP code in this example.

To that end, the toolchain is loaded into the environment: 

$ vpkg_require openmpi/4.1.4:intel-2020
Adding dependency `intel/2020u4` to your environment
Adding dependency `ucx/1.13.1` to your environment
Adding package `openmpi/4.1.4:intel-2020` to your environment

Creating the Package Directory Hierarchy

On DARWIN every user has a home directory in which software can be compiled and stored. For every workgroup to which the user is a member, there is workgroup storage that can also be used for that purpose: if a software package is being maintained for the entire workgroup, it's best to store its variants there. In this example it will be assumed that the software is maintained personally by and for the user.

A directory hierarchy is utilized to organize software packages and versions. The user will be creating versions and variants of VASP, so the top-level directory will be: 

$ VASP_PREFIX="${HOME}/sw/vasp"
$ mkdir -p "$VASP_PREFIX"
$ mkdir "${VASP_PREFIX}/attic"

The attic subdirectory is used to store the source code packages for VASP and any other components associated with that software – so in this case, the Wannier90 source will be placed there, as well. To download the Wannier90 3.1.0 source, for example: 

$ pushd "${VASP_PREFIX}/attic"
$ wget "https://github.com/wannier-developers/wannier90/archive/v3.1.0.tar.gz"
$ popd

This recipe will be creating a variant of VASP 6.3.2 with Wannier90 3.1.0 functionality, so the version of the package will combine those attributes as well as the compiler toolchain used. For the Intel compiler suite, then, the version will be 6.3.2 with attributes intel and wannier90-3.1.0: 

$ VASP_VERSION="6.3.2:wannier90-3.1.0,intel"
$ VASP_INSTALL_PATH="$(vpkg_id2path --package-prefix "$VASP_PREFIX" --version-id "$VASP_VERSION")"
$ echo "$VASP_INSTALL_PATH"
/home/1001/sw/vasp/6.3.2-intel-wannier90-3.1.0
$ mkdir "$VASP_INSTALL_PATH"

The source code will be colocated in that variant directory, so a src subdirectory will be created and the source packages unpacked therein: 

$ mkdir "${VASP_INSTALL_PATH}/src"
$ cd "${VASP_INSTALL_PATH}/src"
$ tar -xf ../attic/v3.1.0.tar.gz
$ tar -xf ../attic/vasp.6.3.2.tar.gz
$ ls -l
total 36858
drwxr-xr-x  4 frey everyone        13 Feb  8 12:17 vasp.6.3.2
drwxr-xr-x 12 frey everyone        28 Feb  8 12:21 wannier90-3.1.0

Building Wannier90

Next, Wannier90 is built. The first step is to copy and modify a make.inc file for the build system: 

$ cd wannier90-3.1.0
$ cp config/make.inc.ifort_tcm make.inc

The make.inc should be modified accordingly:

make.inc
#=====================================================
# For Linux with intel version 11/12 on 64bit machines
#=====================================================
F90 = ifort
COMMS=mpi
MPIF90=mpif90
FCOPTS=-O2
LDOPTS=-O2
 
#========================================================
# Intel mkl libraries. Set LIBPATH if not in default path
#========================================================
 
LIBDIR = 
LIBS   =  -mkl=sequential
 
#=======================
# ATLAS Blas and LAPACK
#=======================
#LIBDIR = /usr/local/lib
#LIBS = -L$(LIBDIR)  -llapack -lf77blas -lcblas -latlas

Note the use of the special Intel compiler flag -mkl which causes the Intel Fortran compiler to emit the necessary library paths and names to compile and link against the MKL library. Were you building with threaded parallelism in the MKL library the flag would instead be -mkl=parallel.

Build is now accomplished using make and the target all: 

$ make all

If successful, the completed library and executables can be installed in the VASP installation directory: 

$ make PREFIX="$VASP_INSTALL_PATH" install
$ ls -l ../../lib
total 1506
-rw-r--r-- 1 frey everyone 3004552 Feb  8 11:52 libwannier.a
 
$ ls -l ../../bin
total 77578
-rwxr-xr-x 1 frey everyone  3119128 Feb  8 11:52 postw90.x
-rwxr-xr-x 1 frey everyone  3507440 Feb  8 11:52 w90chk2chk.x
-rwxr-xr-x 1 frey everyone  1074040 Feb  8 11:52 w90pov
-rwxr-xr-x 1 frey everyone  3483032 Feb  8 11:52 w90spn2spn.x
-rwxr-xr-x 1 frey everyone   988728 Feb  8 11:52 w90vdw.x
-rwxr-xr-x 1 frey everyone  3464584 Feb  8 11:52 wannier90.x

With the Wannier90 library completed, the VASP build can proceed.

Building VASP

The VASP source code is one directory level up; a customized copy of the Intel OpenMP + MKL + MPI architecture-specific file is required: 

$ cd ../vasp.6.3.2
$ cp arch/makefile.include.intel_ompi_mkl_omp makefile.include

The following patch can be downloaded to the vasp.6.3.2 source directory:

makefile.include.patch
--- A/makefile.include	2023-03-13 21:27:27.603020557 -0400
+++ B/makefile.include	2023-03-13 21:27:09.654568807 -0400
@@ -13,7 +13,7 @@
 CPP         = fpp -f_com=no -free -w0  $*$(FUFFIX) $*$(SUFFIX) $(CPP_OPTIONS)
 
 FC          = mpif90 -qopenmp
-FCL         = mpif90
+FCL         = mpif90 -qopenmp
 
 FREE        = -free -names lowercase
 
@@ -49,19 +49,19 @@
 
 # When compiling on the target machine itself, change this to the
 # relevant target when cross-compiling for another architecture
-VASP_TARGET_CPU ?= -xHOST
+VASP_TARGET_CPU ?= -xcore-avx2
 FFLAGS     += $(VASP_TARGET_CPU)
 
 # Intel MKL for FFTW, BLAS, LAPACK, and scaLAPACK
 # (Note: for Intel Parallel Studio's MKL use -mkl instead of -qmkl)
-FCL        += -qmkl
-MKLROOT    ?= /path/to/your/mkl/installation
+FCL        += -mkl=sequential
+MKLROOT    ?= 
 INCS        =-I$(MKLROOT)/include/fftw
 
 # Use a separate scaLAPACK installation (optional but recommended in combination with OpenMPI)
 # Comment out the two lines below if you want to use scaLAPACK from MKL instead
-SCALAPACK_ROOT ?= /path/to/your/scalapack/installation
-LLIBS      += -L${SCALAPACK_ROOT}/lib -lscalapack
+SCALAPACK_ROOT ?=
+LLIBS      += -L$(MKLROOT)/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64
 
 # HDF5-support (optional but strongly recommended)
 #CPP_OPTIONS+= -DVASP_HDF5
@@ -70,9 +70,9 @@
 #INCS       += -I$(HDF5_ROOT)/include
 
 # For the VASP-2-Wannier90 interface (optional)
-#CPP_OPTIONS    += -DVASP2WANNIER90
-#WANNIER90_ROOT ?= /path/to/your/wannier90/installation
-#LLIBS          += -L$(WANNIER90_ROOT)/lib -lwannier
+CPP_OPTIONS    += -DVASP2WANNIER90
+WANNIER90_ROOT ?=
+LLIBS          += -L$(WANNIER90_ROOT)/lib -lwannier
 
 # For the fftlib library (hardly any benefit in combination with MKL's FFTs)
 #CPP_OPTION += -Dsysv

and applied to the copy of the stock makefile.include: 

$ patch -p1 < makefile.include.patch
patching file makefile.include

In summary, the changes embodied in that patch are:

  • When the object code is linked using the $(FCL) command OpenMP libraries are only included if the -qopenmp flag is present
  • The AMD nodes in DARWIN have AVX2 capabilities but the -xHOST may not detect it; using -xcore-avx2 forces AVX2 optimizations
  • The -qmkl flag isn't accepted by pre-2022 Intel suites and would default to the threaded library; use -mkl=sequential instead
  • $(MKLROOT) is present in the Intel suite runtime environment so no override value is necesssary
  • The Intel MKL ScaLAPACK and BLACS libraries are found under the $(MKLROOT) directory so a path relative to that is appropriate
  • The appropriate integer bit-depth libraries (lp64) with BLACS for Open MPI runtimes are needed
  • The Wannier90 library doesn't require a hard-coded path, we'll provide that from the environment; also, the Wannier90 flags should be uncommented

The build of VASP can now be effected: 

$ make WANNIER90_ROOT="$VASP_INSTALL_PATH"

By default all three variants of VASP are produced – std, gam, and ncl. If the build is successful, the completed executables can be copied into the installation directory: 

$ mkdir "${VASP_INSTALL_PATH}/bin"
$ cp bin/* "${VASP_INSTALL_PATH}/bin"

At this point all executables associated with Wannier90 and VASP are present in the ${VASP_INSTALL_PATH}/bin directory and the Wannier90 library is present in ${VASP_INSTALL_PATH}/lib. To make use of this variant of VASP, the ${VASP_INSTALL_PATH}/bin directory must be added to the PATH environment variable. That is where VALET comes in.

VALET Package Definition

Since this copy of VASP resides in the user's home directory, the appropriate location to install a VALET package definition for it is ~/.valet. This directory is present by default for all DARWIN users.

Key values to know before editing the package definition were assigned to variables throughout this recipe: 

$ for V in VASP_VERSION VASP_PREFIX; do echo "$V = $(eval echo \$$V)"; done
VASP_VERSION = 6.3.2:wannier90-3.1.0,intel
VASP_PREFIX = /home/1001/sw/vasp
 
$ vpkg_history
[standard]
  intel/2020u4
  ucx/1.13.1
  openmpi/4.1.4:intel-2020

Recall that we added openmpi/4.1.4:intel-2020 to the environment with vpkg_require; that is the single direct dependency for this variant of VASP. The following template can be saved to ~/.valet/vasp.vpkg_yaml and edited to include the values of those variables:

vasp.vpkg_yaml
vasp:
    prefix: $VASP_PREFIX
    description: Vienna Ab initio Simulation Package
    url: http://www.vasp.at
    
    default-version: $VASP_VERSION
    versions:
        "$VASP_VERSION":
            description: VASP 6.3.2, Wannier90 3.1.0, Intel compilers and MKL
            dependencies:
                - openmpi/4.1.4:intel-2020

VALET can now be used to query versions of VASP available: 

$ vpkg_versions vasp
 
Available versions/variants in package (* = default version):
 
[/home/1001/.valet/vasp.vpkg_yaml]
vasp                           Vienna Ab initio Simulation Package
* 6.3.2:intel,wannier90-3.1.0  VASP 6.3.2, Wannier90 3.1.0, Intel compilers and MKL

Using VASP

Henceforth, this variant of VASP 6.3.2 can be added to your shell environment using VALET: 

$ vpkg_require vasp/6.3.2:wannier90-3.1.0,intel
Adding dependency `intel/2020u4` to your environment
Adding dependency `ucx/1.13.1` to your environment
Adding dependency `openmpi/4.1.4:intel-2020` to your environment
Adding package `vasp/6.3.2:wannier90-3.1.0,intel` to your environment
 
$ which vasp_std
/home/1001/sw/vasp/6.3.2-intel-wannier90-3.1.0/bin/vasp_std
 
$ which wannier90.x
/home/1001/sw/vasp/6.3.2-intel-wannier90-3.1.0/bin/wannier90.x

The same vpkg_require command can be used in a job script to setup the runtime environment for the execution of VASP. The Open MPI job script template at /opt/templates/slurm/generic/mpi/openmpi/openmpi.qs can be copied and modified accordingly: 

   :
#
# [EDIT] Do any pre-processing, staging, environment setup with VALET
#        or explicit changes to PATH, LD_LIBRARY_PATH, etc.
#
vpkg_require vasp/6.3.2:wannier90-3.1.0,intel
   :
#
# [EDIT] Execute your MPI program
#
${UD_MPIRUN} vasp_std 
mpi_rc=$?
 
#
# [EDIT] Do any cleanup work here...
#
 
#
# Be sure to return the mpirun's result code:
#
exit $mpi_rc
  • technical/recipes/vasp-wannier.txt
  • Last modified: 2023-03-14 15:17
  • by anita