software:wrf:caviness

WRF on Caviness

Currently WRF is not installed on Caviness by the system administrators. It can be installed by users for personal use or for use by their workgroups by following the below directions. These directions will walk users through all the steps required to install and compile WRF and WPS. It will also explain how to add the WRF installation into your home or workgroup's VALET directory. Lastly, these steps will walk you through a trial run.

These directions are loosely based on the directions from NCAR's "How to compile WRF: The Complete Process" page. Our directions differ from NCAR's in order to get better performance for WRF on Caviness. The main difference is the default MPI software WRF leverages, MPICH, whereas Open MPI is the MPI software tightly-integrated with Slurm on Caviness. These directions will take you through the steps of installing WRF to use Open MPI instead of MPICH for parallel processing.

If this is the first time that someone is installing software in your workgroup it is suggested that your follow the directory structure similar to the it_css workgroup by using /work/it_css/sw/ and substituting your workgroup. Not only will it make it easier to follow these directions, but it will also make it easier to add more software packages and versions in the future.

The directions below will walk you through the download, installing, compiling WRF, and creating a WRF VALET package. Make sure you follow the directions CAREFULLY and make the necessary changes to reflect your personal environment. The directions and examples below are based on user traine in workgroup it_css. You will need to change all references to reflect your username and workgroup.

Start by joining your workgroup

  [traine@login01 ~]$ workgroup -g it_css
  [(it_css:traine)@login01 ~]$

The next commands load the required VALET packages and set the necessary environment variables.

Use the VALET commmand vpkg_require to load the appropriate version of the openmpi package.

 [(it_css:traine)@login01 ~]$ vpkg_require openmpi/4.0.2:intel
  Adding dependency `intel/2019u5` to your environment
  Adding dependency `libfabric/1.9.0` to your environment
  Adding package `openmpi/4.0.2:intel` to your environment

Set the environment variables needed for compiling and installing the WRF packages.

  [(it_css:traine)@login01 ~]$ WRF_PREFIX=$WORKDIR/sw/wrf/20210121 ## MAKE SURE THAT THIS IS CHANGED TO YOUR WORK GROUP DIRECTORY OR HOME DIRECTORY ##
  [(it_css:traine)@login01 ~]$ mkdir -p "$WRF_PREFIX"
  [(it_css:traine)@login01 ~]$ WRF_SRC="${WRF_PREFIX}/src"
  [(it_css:traine)@login01 ~]$ WRF_BIN="${WRF_PREFIX}/bin"
  [(it_css:traine)@login01 ~]$ WRF_INC="${WRF_PREFIX}/include"
  [(it_css:traine)@login01 ~]$ WRF_LIB="${WRF_PREFIX}/lib"
  [(it_css:traine)@login01 ~]$ WRF_LIBRARIES_SRC="${WRF_SRC}/LIBRARIES"
  [(it_css:traine)@login01 ~]$ mkdir -p "$WRF_LIBRARIES_SRC"
  [(it_css:traine)@login01 ~]$ WRF_TESTS_SRC="${WRF_SRC}/TESTS"
  [(it_css:traine)@login01 ~]$ mkdir -p "$WRF_TESTS_SRC"
  [(it_css:traine)@login01 ~]$ export CC=icc
  [(it_css:traine)@login01 ~]$ export CFLAGS="-xHost"
  [(it_css:traine)@login01 ~]$ export CXX=icpc
  [(it_css:traine)@login01 ~]$ export CXXFLAGS="-xHost"
  [(it_css:traine)@login01 ~]$ export CPPFLAGS="-I${WRF_INC}"
  [(it_css:traine)@login01 ~]$ export FC=ifort
  [(it_css:traine)@login01 ~]$ export FCFLAGS="-I${WRF_INC} -xHost" 
  [(it_css:traine)@login01 ~]$ export F77="$FC"
  [(it_css:traine)@login01 ~]$ export LDFLAGS="-L${WRF_LIB}"
  [(it_css:traine)@login01 ~]$ export PATH="${WRF_BIN}:$PATH"
  [(it_css:traine)@login01 ~]$ export MPICC="mpicc"
  [(it_css:traine)@login01 ~]$ export MPICXX="mpicxx"
  [(it_css:traine)@login01 ~]$ export MPIFC="mpifort"

WRF requires several libraries for the installation. First, we use the environment variable WRF_LIBRARIES_SRC to change into the LIBRARIES directory, then we will use the wget command to download the libraries, and finally use the appropriate commands to install each library.

Download libraries

  [(it_css:traine)@login01 ~]$ cd "$WRF_LIBRARIES_SRC"
  [(it_css:traine)@login01 LIBRARIES]$ wget https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compile_tutorial/tar_files/netcdf-4.1.3.tar.gz
  [(it_css:traine)@login01 LIBRARIES]$ wget https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compile_tutorial/tar_files/jasper-1.900.1.tar.gz
  [(it_css:traine)@login01 LIBRARIES]$ wget https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compile_tutorial/tar_files/libpng-1.2.50.tar.gz
  [(it_css:traine)@login01 LIBRARIES]$ wget https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compile_tutorial/tar_files/zlib-1.2.7.tar.gz

Install netcdf library

  [(it_css:traine)@login01 LIBRARIES]$ tar -xf netcdf-4.1.3.tar.gz
  [(it_css:traine)@login01 LIBRARIES]$ cd netcdf-4.1.3
  [(it_css:traine)@login01 netcdf-4.1.3]$ ./configure --prefix="$WRF_PREFIX" --disable-dap --disable-netcdf-4 --disable-shared
  [(it_css:traine)@login01 netcdf-4.1.3]$ make -j 4
  [(it_css:traine)@login01 netcdf-4.1.3]$ make install
  [(it_css:traine)@login01 netcdf-4.1.3]$ cd ..

Install zlib library

  [(it_css:traine)@login01 LIBRARIES]$ tar -xf zlib-1.2.7.tar.gz
  [(it_css:traine)@login01 LIBRARIES]$ cd zlib-1.2.7
  [(it_css:traine)@login01 zlib-1.2.7]$ ./configure --prefix="$WRF_PREFIX"
  [(it_css:traine)@login01 zlib-1.2.7]$ make
  [(it_css:traine)@login01 zlib-1.2.7]$ make install
  [(it_css:traine)@login01 zlib-1.2.7]$ cd ..

Install libpng library

  [(it_css:traine)@login01 LIBRARIES]$ tar -xf libpng-1.2.50.tar.gz
  [(it_css:traine)@login01 LIBRARIES]$ cd libpng-1.2.50
  [(it_css:traine)@login01 libpng-1.2.50]$ ./configure --prefix="$WRF_PREFIX"
  [(it_css:traine)@login01 libpng-1.2.50]$ make -j 4
  [(it_css:traine)@login01 libpng-1.2.50]$ make install
  [(it_css:traine)@login01 libpng-1.2.50]$ cd ..

Install jasper library

  [(it_css:traine)@login01 LIBRARIES]$ tar -xf jasper-1.900.1.tar.gz
  [(it_css:traine)@login01 LIBRARIES]$ cd jasper-1.900.1
  [(it_css:traine)@login01 jasper-1.900.1]$ ./configure --prefix="$WRF_PREFIX"
  [(it_css:traine)@login01 jasper-1.900.1]$ make -j 4
  [(it_css:traine)@login01 jasper-1.900.1]$ make install
  [(it_css:traine)@login01 jasper-1.900.1]$ cd ..

Use the environment variable WRF_TEST_SRC to change into the TESTS directory. Next download the test file, compile them using the libraries you installed and verify the results. Remember all executables should typically be run on compute nodes. However these tests require compiling which must be done either on the login (head) node or by using salloc --partition=devel to run on compute nodes, and since these tests will run quickly and only use minimal resources to verify the correct installation of the libraries, we will use the login (head) node for this demonstration.

  [(it_css:traine)@login01 LIBRARIES]$ cd "$WRF_TESTS_SRC"
  [(it_css:traine)@login01 TESTS]$ 

Download and extract test file

  [(it_css:traine)@login01 TESTS]$ wget https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compile_tutorial/tar_files/Fortran_C_NETCDF_MPI_tests.tar
  [(it_css:traine)@login01 TESTS]$ tar -xf Fortran_C_NETCDF_MPI_tests.tar

Compile and run test (serial)

  [(it_css:traine)@login01 TESTS]$ $FC $FCFLAGS -c 01_fortran+c+netcdf_f.f $LDFLAGS
  [(it_css:traine)@login01 TESTS]$ $CC $CPPFLAGS -c 01_fortran+c+netcdf_c.c $CFLAGS $LDFLAGS
  [(it_css:traine)@login01 TESTS]$ $FC -c 01_fortran+c+netcdf_f.f $FCFLAGS $LDFLAGS
  [(it_css:traine)@login01 TESTS]$ $CC $CPPFLAGS -c 01_fortran+c+netcdf_c.c $CFLAGS $LDFLAGS
  [(it_css:traine)@login01 TESTS]$ $FC 01_fortran+c+netcdf_f.o 01_fortran+c+netcdf_c.o $FCFLAGS $LDFLAGS -lnetcdff -lnetcdf
  [(it_css:traine)@login01 TESTS]$ ./a.out
Verify your test results. This can be done by comparing them to the test results on the NCAR site. The results are in the Library Compatibility Tests section Test #1

Compile and run test (MPI)

  [(it_css:traine)@login01 TESTS]$ $MPICC $CPPFLAGS -c 02_fortran+c+netcdf+mpi_c.c $CFLAGS $LDFLAGS
  [(it_css:traine)@login01 TESTS]$ $MPIFC -c 02_fortran+c+netcdf+mpi_f.f $FCFLAGS $LDFLAGS
  [(it_css:traine)@login01 TESTS]$ $MPICC $CPPFLAGS -c 02_fortran+c+netcdf+mpi_c.c $CFLAGS $LDFLAGS
  [(it_css:traine)@login01 TESTS]$ $MPIFC 02_fortran+c+netcdf+mpi_f.o 02_fortran+c+netcdf+mpi_c.o $FCFLAGS $LDFLAGS -lnetcdff -lnetcdf
  [(it_css:traine)@login01 TESTS]$ mpirun -np 2 ./a.out
Verify your test results. This can be done by comparing them to the test results on the NCAR site. The results are in the Library Compatibility Tests section: Test #2

Download WRF source code

  [(it_css:traine)@login01 TESTS]$ cd "$WRF_SRC"
  [(it_css:traine)@login01 src]$ wget https://www2.mmm.ucar.edu/wrf/src/WRFV4.0.TAR.gz
  [(it_css:traine)@login01 src]$ tar -xf WRFV4.0.TAR.gz
  [(it_css:traine)@login01 src]$ cd WRF
  [(it_css:traine)@login01 WRF]$ 
The compile steps can take quite a long time to complete. Use caution and make sure that you select the correct options as directed below.

Configure WRF serial build

Configuring serial build. You will want to select option 64 and nesting option 0.

  [(it_css:traine)@login01 WRF]$ export NETCDF="$WRF_PREFIX"
  [(it_css:traine)@login01 WRF]$ ./configure
  [(it_css:traine)@login01 WRF]$ ./compile -j 4 em_real
  [(it_css:traine)@login01 WRF]$ makedir -p "$WRF_BIN"
  [(it_css:traine)@login01 WRF]$ install --target-directory="$WRF_BIN" --mode=0775 main/*.exe

Configure WRF parallel build

Configuring parallel build. You will want to select option 66 and nesting option 1.

  [(it_css:traine)@login01 WRF]$  ./clean -a 
  [(it_css:traine)@login01 WRF]$ ./configure

Apply required patch for OpenMPI flags

The patch command with the use of EOT does not display the bash prompt after it is entered, but instead >, because it is waiting for the patch lines to be entered at the terminal until EOT is typed. You may find it best to copy and paste the lines below to avoid making any typing mistakes as spacing is very important and it must be typed exactly done this way..

  [(it_css:traine)@login01 WRF]$ patch -p1 <<EOT
  --- A/configure.wrf 2020-12-10 14:06:01.907649095 -0500
  +++ B/configure.wrf 2020-12-10 14:40:00.791338460 -0500
  @@ -118,8 +118,8 @@
   SFC             =       ifort
   SCC             =       icc
   CCOMP           =       icc
  -DM_FC           =       mpif90 -f90=\$(SFC)
  -DM_CC           =       mpicc -cc=\$(SCC)
  +DM_FC           =       mpif90
  +DM_CC           =       mpicc
   FC              =       time \$(DM_FC)
   CC              =       \$(DM_CC) -DFSEEKO64_OK 
   LD              =       \$(FC)
  @@ -140,7 +140,7 @@
   BYTESWAPIO      =       -convert big_endian
   RECORDLENGTH    =       -assume byterecl
   FCBASEOPTS_NO_G =       -ip -fp-model precise -w -ftz -align all -fno-alias \$(FORMAT_FREE) \$(BYTESWAPIO) -xHost -fp-model fast=2 -no-heap-arrays -no-prec-div -no-prec-sqrt -fno-common -xCORE-AVX2
  -FCBASEOPTS      =       \$(FCBASEOPTS_NO_G) \$(FCDEBUG)
  +FCBASEOPTS      =       \$(FCBASEOPTS_NO_G) \$(FCDEBUG) -assume nocc_omp
   MODULE_SRCH_FLAG =     
   TRADFLAG        =      -traditional-cpp
   CPP             =      /lib/cpp -P -nostdinc
  EOT

Compile WRF parallel build

  [(it_css:traine)@login01 WRF]$ ./compile -j 4 em_real
  [(it_css:traine)@login01 WRF]$ mkdir -p "$WRF_BIN"
  [(it_css:traine)@login01 WRF]$ for exe in main/*.exe; do
     # Install each exe to WRF_BIN with the prefix "mpi_" on it
     # to differentiate from the serial variants:
     WRF_ROOT="$(echo $exe | sed -e 's/main\///')"
     install --mode=0775 "$exe" "${WRF_BIN}/mpi_${WRF_ROOT}"
   done
  [(it_css:traine)@login01 WRF]$

Download WPS source code

  [(it_css:traine)@login01 WRF]$ cd "$WRF_SRC"
  [(it_css:traine)@login01 src]$ wget https://www2.mmm.ucar.edu/wrf/src/WPSV4.0.TAR.gz
  [(it_css:traine)@login01 src]$ tar -xf WPSV4.0.TAR.gz
  [(it_css:traine)@login01 src]$ cd WPS
  [(it_css:traine)@login01 WPS]$ 
  • WPS unlike WRF only has a serial build and currently does not have parallel build.
  • The compile steps can take quite a long time to complete. Use caution and make sure that you select the correct options as directed below.

Configure and compile WPS serial build

Configure serial build. You will want to select option 17.

  [(it_css:traine)@login01 WPS]$ export JASPERLIB="$WRF_LIB"
  [(it_css:traine)@login01 WPS]$ export JASPERINC="$WRF_INC"
  [(it_css:traine)@login01 WPS]$ ./configure
  [(it_css:traine)@login01 WPS]$ ./compile
  [(it_css:traine)@login01 WPS]$ install --target-directory="$WRF_BIN" --mode=0775 *.exe

Now select option 19

  [(it_css:traine)@login01 WPS]$ ./clean -a 
  [(it_css:traine)@login01 WPS]$ ./configure
  [(it_css:traine)@login01 WPS]$ ./compile
  [(it_css:traine)@login01 WPS]$ for exe in *.exe; do
    # Install each exe to WRF_BIN with the prefix "mpi_" on it
    # to differentiate from the serial variants:
    install --mode=0775 "$exe" "${WRF_BIN}/mpi_${exe}"
  done

If you are planning on being the only person running WRF or are installing a custom version of WRF it is suggested that you add the WRF VALET package to the $HOME/.valet directory. If you are installing WRF and planning for multiple people in your workgroup to run it, then you should add the WRF VALET package to the $WORKDIR/sw/valet directory. You do not have to add it in both places.

If you have done this before then you just need to append the new version to the extant VALET package definition file.

Adding WRF VALET package in HOME directory

  [(it_css:traine)@login01 WRF]$ mkdir -p ~/.valet       #This step is only required if you don't have a .valet directory already created
  [(it_css:traine)@login01 WRF]$ cat <<EOT > ~/.valet/wrf.vpkg_yaml
   #
   # WRF - Weather Research and Forecasting model
   #
   wrf:
       prefix: $(dirname "$WRF_PREFIX")
       description: Weather Research and Forecasting model
       url: "https://www.mmm.ucar.edu/weather-research-and-forecasting-model"
       versions:
          "$(basename "$WRF_PREFIX")":
              description: 2020-12-10 build with Intel compilers and Open MPI
              dependencies:
                  - openmpi/4.0.2:intel
 
   EOT
   [(it_css:traine)@login01 WRF]$

Adding WRF VALET package in workgroup directory

As mentioned at the top of this page these directions are based on user traine in workgroup it_css. In these next directions we are going to add the WRF VALET package in the it_css workgroup directory. The location of the VALET directory might differ in your workgroup directory, or you might not yet have a VALET directory in your workgroup. If you do not have a valet directory, we suggest that you follow the directory structure we use in the it_css workgroup. Not only will it make it easier to follow these directions, but it will also make it easy to add more software packages and versions in the future.

  [(it_css:traine)@login01 ~]$ cd /work/it_css/sw
  [(it_css:traine)@login01 sw]$ mkdir -p valet           #This step is only required if you don't have a valet directory already created
  [(it_css:traine)@login01 sw]$ cat <<EOT > valet/wrf.vpkg_yaml
   #
   # WRF - Weather Research and Forecasting model
   #
   wrf:
       prefix: $(dirname "$WRF_PREFIX")
       description: Weather Research and Forecasting model
       url: "https://www.mmm.ucar.edu/weather-research-and-forecasting-model"
       versions:
          "$(basename "$WRF_PREFIX")":
              description: 2020-12-10 build with Intel compilers and Open MPI
              dependencies:
                  - openmpi/4.0.2:intel
 
   EOT
  [(it_css:traine)@login01 sw]$

If all the prior steps were followed and successfully completed you are now ready to check if you can successfully load WRF with VALET and then find the WRF executables.

Clear environment and load WRF VALET package

  [(it_css:traine)@login01 ~]$ vpkg_rollback all
  [(it_css:traine)@login01 ~]$ vpkg_require wrf
  Adding dependency `intel/2019u5` to your environment
  Adding dependency `libfabric/1.9.0` to your environment
  Adding dependency `openmpi/4.0.2:intel` to your environment
  Adding package `wrf/20210121` to your environment

Check WRF executables

  [(it_css:traine)@login01 ~]$ which real.exe
  /work/it_css/sw/WRF/20210113/bin/real.exe
  [(it_css:traine)@login01 ~]$ which mpi_real.exe
  /work/it_css/sw/WRF/20210113/bin/mpi_real.exe

By default WRF and WPS are setup to run in their respected SRC directories. This is not the best idea and can cause serious problems to the WRF installation and also block the WRF software from being run by multiple people at the same time. The following steps will show you how to set up a safe environment to run WRF and WPS from a directory outside the SRC directories.

By default WPS/WRF is set up to run from it's installed SRC directory. This is not ideal, so running WPS/WRF from another directory outside it's install directory is much safer and can help protect against accidental modifications or deletions of the originally installed source files. To set up the WPS and WRF to run their respected programs outside their installed SRC directory we have created an example to test called example_wrf.tar available in trainf's home directory to copy into the directory location in which you plan to download the data files and run all the WRF programs. For this demo, the example_wrf.tar file will be copied into the location /work/it_css/traine/wrf.

  [(it_css:traine)@login01 wrf]$ cp /home/1200/wrf/example_wrf.tar .

Now that you have the example_wrf.tar file where you want it, now we can extract it and and start to set up the WPS programs.

  [(it_css:traine)@login01 wrf]$ tar -xvf example_wrf.tar
  example_wrf/
  example_wrf/wrf_job/
  example_wrf/wrf_job/Makefile
  example_wrf/wrf_job/wrfmpi.qs
  example_wrf/wrf_job/namelist.input
  example_wrf/data/
  example_wrf/data/WPS_GEOG/
  example_wrf/data/GFS/
  example_wrf/data/GFS/modelDataDownload.sh
  example_wrf/data/wps_geo_download.sh
  example_wrf/wps_job/
  example_wrf/wps_job/namelist.wps
  example_wrf/wps_job/Makefile

Now you need to download the Statis Geography Data and uncompress it by using the provided script wps_geo_download.sh provided in this example. This is a fairly large set of files (~29GB) and could take some download and uncompress.

  [(it_css:traine)@login01 wrf]$ cd example_wrf/data
  [(it_css:traine)@login01 data]$ ./wps_geo_download.sh
  ...MANY LINES AND MINUTES LATER...

If this is successful, then it is suggested that you remove the geog_high_res_mandatory.tar file.

  [(it_css:traine)@login01 data]$ rm geog_high_res_mandatory.tar

Now we will download some GFS model data. You will need to update the modelDataDownload.sh script provided in the example to the appropriate dates you want for the data available from NOAA's FTP site. The directory should be formatted something like gfs.YYYYMMDD. In this example, we will be using gfs.20210107.

  [(it_css:traine)@login01 data]$ cd GFS

In the modelDataDownload.sh file you will want to update the yyyy, mm, dd, hh, fff, and model to match the date and model data that you are trying to download. This example used 01/07/2021 for the 00Z run for the GFS model.

This data is not longer available on NOAA's FTP website, so the example below will not work if you copy it directly and do not change the dates.

After making the necessary changes, you can run modelDataDownload.sh script. To do this you will want to use sbatch and the download.qs script. The download.qs will run the modelDataDownload.sh script on a compute node to download all the requested model data.

  [(it_css:traine)@login01 GFS]$ sbatch download.qs
  [(it_css:traine)@login01 GFS]$ ls 
  gfs_20210107_00z_000f  gfs_20210107_00z_078f  gfs_20210107_00z_156f  gfs_20210107_00z_234f  gfs_20210107_00z_312f
  gfs_20210107_00z_003f  gfs_20210107_00z_081f  gfs_20210107_00z_159f  gfs_20210107_00z_237f  gfs_20210107_00z_315f
  gfs_20210107_00z_006f  gfs_20210107_00z_084f  gfs_20210107_00z_162f  gfs_20210107_00z_240f  gfs_20210107_00z_318f
  gfs_20210107_00z_009f  gfs_20210107_00z_087f  gfs_20210107_00z_165f  gfs_20210107_00z_243f  gfs_20210107_00z_321f
  gfs_20210107_00z_012f  gfs_20210107_00z_090f  gfs_20210107_00z_168f  gfs_20210107_00z_246f  gfs_20210107_00z_324f
  ...                    ...                    ...                    ...                    ...
  gfs_20210107_00z_072f  gfs_20210107_00z_150f  gfs_20210107_00z_228f  gfs_20210107_00z_306f  gfs_20210107_00z_384f
  gfs_20210107_00z_075f  gfs_20210107_00z_153f  gfs_20210107_00z_231f  gfs_20210107_00z_309f

To prevent an issue later down the road, we will need to move the modelDataDownload.sh script up one directory into the data directory.

  [(it_css:traine)@login01 GFS]$ mv modelDataDownload.sh download.qs ../

Now we have the required metadata files and sample GFS data downloaded to run WPS and WRF.

To avoid running executable on the login node, we are going to request a interactive compute node session using salloc.

  [(it_css:traine)@login01 GFS]$ salloc --partition=it_css
  salloc: Granted job allocation 11715984
  salloc: Waiting for resource configuration
  salloc: Nodes r00n49 are ready for job
  [triane@r00n49 GFS]$  

Now we are ready to process the downloaded GFS data with the WPS scripts for our example. To do this we will need to change directories into the the wps_job directory, then use of the make command to gather the require files that you will need to run WPS.

  [triane@r00n49 GFS]$  cd ../../wps_job/
  [triane@r00n49 wps_job]$  make
  cp /work/it_css/sw/WRF/20210113/src/WPS/geogrid/GEOGRID.TBL GEOGRID.TBL
  cp /work/it_css/sw/WRF/20210113/src/WPS/metgrid/METGRID.TBL METGRID.TBL
  cp /work/it_css/sw/WRF/20210113/src/WPS/ungrib/Variable_Tables/Vtable.GFS Vtable
  cp /work/it_css/sw/WRF/20210113/src/WPS/link_grib.csh link_grib.csh

After that is completed you will need to update the namelist.wps file to reflect the model data that you downloaded in the prior steps. Use nano or vim and make the necessary changes noted with #EDIT THIS LINE and #ADD THIS LINE.

  &share
   wrf_core = 'ARW',
   max_dom = 1,
   start_date = '2021-01-07_00:00:00','2006-08-16_12:00:00', #EDIT THIS LINE
   end_date   = '2021-01-23_23:00:00','2006-08-16_12:00:00', #EDIT THIS LINE
   interval_seconds = 10800 #edit this line
   io_form_geogrid = 2,
  /
  &geogrid
   parent_id         =   1,   1,
   parent_grid_ratio =   1,   3,
   i_parent_start    =   1,  31,
   j_parent_start    =   1,  17,
   e_we              =  10, 112, #EDIT THIS LINE
   e_sn              =  10,  97, #EDIT THIS LINE
  ...
   geog_data_res = 'default','default',
   dx = 10000, #edit this line
   dy = 10000, #edit this line
   map_proj = 'lambert',
   ref_lat   =  38.72, #edit this line
   ref_lon   = -75.08, #edit this line
   truelat1  =  38.72, #edit this line
   truelat2  =  38.72, #edit this line 
   stand_lon = -75.08, #edit this line
   geog_data_path = '../data/WPS_GEOG/', #EDIT THIS LINE
   opt_geogrid_tbl_path = './' #OPTIONAL EDIT THIS LINE
   ...
  &metgrid
   fg_name = 'FILE'
   io_form_metgrid = 2,
   opt_metgrid_tbl_path = './' #ADD THIS LINE
  /

Now you are ready to run geogrid.exe

  [triane@r00n49 wps_job]$ geogrid.exe
  Parsed 28 entries in GEOGRID.TBL
  Processing domain 1 of 1
  Processing XLAT and XLONG
  Processing MAPFAC
  Processing F and E
  Processing ROTANG
  Processing LANDUSEF
  Calculating landmask from LANDUSEF ( WATER =  17  21 )
  Processing HGT_M
  Processing SOILTEMP
  Processing SOILCTOP
  Processing SCT_DOM
  Processing SOILCBOT
  Processing SCB_DOM
  Processing ALBEDO12M
  Processing GREENFRAC
  Processing LAI12M
  Processing SNOALB
  Processing CON
  Processing VAR
  Processing OA1
  Processing OA2
  Processing OA3
  Processing OA4
  Processing OL1
  Processing OL2
  Processing OL3
  Processing OL4
  Processing VAR_SSO
 
  Optional fields not processed by geogrid:
    LAKE_DEPTH (priority=1, resolution='default', path='../data/WPS_GEOG/lake_depth/')
    URB_PARAM (priority=1, resolution='default', path='../data/WPS_GEOG/NUDAPT44_1km/')
    FRC_URB2D (priority=1, resolution='default', path='../data/WPS_GEOG/urbfrac_nlcd2011/')
    IMPERV (priority=1, resolution='default', path='../data/WPS_GEOG/nlcd2011_imp_ll_9s/')
    CANFRA (priority=1, resolution='default', path='../data/WPS_GEOG/nlcd2011_can_ll_9s/')
    EROD (priority=1, resolution='default', path='../data/WPS_GEOG/erod/')
    CLAYFRAC (priority=1, resolution='default', path='../data/WPS_GEOG/clayfrac_5m/')
    SANDFRAC (priority=1, resolution='default', path='../data/WPS_GEOG/sandfrac_5m/')
 
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  !  Successful completion of geogrid.        !
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Next run the link_grib.csh script and make sure to pass the correct path to where you are storing the downloaded GFS data.

  [triane@r00n49 wps_job]$  ./link_grib.csh ../data/GFS/

This script will create soft links to the GFS data files into the current directory. You should see something like this

  [triane@r00n49 wps_job]$  ls -l GRIB*
  lrwxrwxrwx 1 traine it_css 33 Jan 26 13:59 GRIBFILE.AAA -> ../data/GFS/gfs_20210107_00z_000f
  lrwxrwxrwx 1 traine it_css 33 Jan 26 13:59 GRIBFILE.AAB -> ../data/GFS/gfs_20210107_00z_003f
  lrwxrwxrwx 1 traine it_css 33 Jan 26 13:59 GRIBFILE.AAC -> ../data/GFS/gfs_20210107_00z_006f
  lrwxrwxrwx 1 traine it_css 33 Jan 26 13:59 GRIBFILE.AAD -> ../data/GFS/gfs_20210107_00z_009f
  lrwxrwxrwx 1 traine it_css 33 Jan 26 13:59 GRIBFILE.AAE -> ../data/GFS/gfs_20210107_00z_012f
  lrwxrwxrwx 1 traine it_css 33 Jan 26 13:59 GRIBFILE.AAF -> ../data/GFS/gfs_20210107_00z_015f
  lrwxrwxrwx 1 traine it_css 33 Jan 26 13:59 GRIBFILE.AAG -> ../data/GFS/gfs_20210107_00z_018f
  lrwxrwxrwx 1 traine it_css 33 Jan 26 13:59 GRIBFILE.AAH -> ../data/GFS/gfs_20210107_00z_021f
  ...

Next run ungrib.exe

  [triane@r00n49 wps_job]$ ungrib.exe

Depending on the amount of data, this could take several minutes. If successful, then you should get a message like this at the end.

  ... #MANY LINES & MINUTES LATER
  **********
  Done deleting temporary files.
  **********
 
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  !  Successful completion of ungrib.   !
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

The last step is to run metgrid.exe

  [triane@r00n49 wps_job]$ metgrid.exe
  ... #MANY LINES & MINUTES LATER
  Processing 2021-01-13_21
    FILE
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  !  Successful completion of metgrid.  !
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Now you can end the salloc session and release the compute node

  [triane@r00n49 wps_job]$ exit
  exit
  [(it_css:traine)@login01 wps_job]$

Now you are ready to set up WRF.

If you have not followed all the prior steps leading up to this point YOU MUST go back and complete them. The WPS steps to download and process files in them are required for the WRF steps to work. After you have completed the prior steps, then you can go into the wrf_job directory.

  [(it_css:traine)@login01 wps_job]$ ../wrf_job
  [(it_css:traine)@login01 wrf_job]$ pwd
  /work/it_css/sw/WRF/example_wrf/wrf_job

Now run the make command. This will copy all the required files needed to run WRF.

  [(it_css:traine)@login01 wrf_job]$ make 
  cp /work/it_css/sw/WRF/20210113/src/WRF/run/GENPARM.TBL GENPARM.TBL
  cp /work/it_css/sw/WRF/20210113/src/WRF/run/HLC.TBL HLC.TBL
  cp /work/it_css/sw/WRF/20210113/src/WRF/run/LANDUSE.TBL LANDUSE.TBL
  cp /work/it_css/sw/WRF/20210113/src/WRF/run/MPTABLE.TBL MPTABLE.TBL
  cp /work/it_css/sw/WRF/20210113/src/WRF/run/SOILPARM.TBL SOILPARM.TBL
  cp /work/it_css/sw/WRF/20210113/src/WRF/run/URBPARM.TBL URBPARM.TBL
  cp /work/it_css/sw/WRF/20210113/src/WRF/run/URBPARM_UZE.TBL URBPARM_UZE.TBL
  cp /work/it_css/sw/WRF/20210113/src/WRF/run/VEGPARM.TBL VEGPARM.TBL
  cp /work/it_css/sw/WRF/20210113/src/WRF/run/ozone.formatted ozone.formatted
  cp /work/it_css/sw/WRF/20210113/src/WRF/run/ozone_lat.formatted ozone_lat.formatted
  cp /work/it_css/sw/WRF/20210113/src/WRF/run/ozone_plev.formatted ozone_plev.formatted
  cp /work/it_css/sw/WRF/20210113/src/WRF/run/RRTM_DATA RRTM_DATA
  cp /work/it_css/sw/WRF/20210113/src/WRF/run/RRTM_DATA_DBL RRTM_DATA_DBL
  cp /work/it_css/sw/WRF/20210113/src/WRF/run/RRTMG_LW_DATA RRTMG_LW_DATA
  cp /work/it_css/sw/WRF/20210113/src/WRF/run/RRTMG_LW_DATA_DBL RRTMG_LW_DATA_DBL
  cp /work/it_css/sw/WRF/20210113/src/WRF/run/RRTMG_SW_DATA RRTMG_SW_DATA
  cp /work/it_css/sw/WRF/20210113/src/WRF/run/RRTMG_SW_DATA_DBL RRTMG_SW_DATA_DBL

Next copy the met_em files that were created in the WPS steps.

    [(it_css:traine)@login01 wrf_job]$ cp ../wps_job/met_em* . 

Now you should be ready to run WRF.

In this example, we are running the WRF real.exe and wrf.exe in WRF/run. These steps should be similar to the steps required to run WRF in /WRF/test/em_real, but there might be some differences that are not covered in these directions.

First, changes need to be made to the namelist.input file by using nano or vim. These changes should reflect the information from the model data that you downloaded earlier. See all lines marked with #EDIT THIS LINE and change accordingly.

  &time_control
  run_days                            = 0,
  run_hours                           = 12,
  run_minutes                         = 0,
  run_seconds                         = 0,
  start_year                          = 2021, 2000, 2000, #EDIT THIS LINE
  start_month                         = 01,   01,   01,   #EDIT THIS LINE
  start_day                           = 07,   24,   24,   #EDIT THIS LINE
  start_hour                          = 00,   12,   12,   #EDIT THIS LINE
  end_year                            = 2021, 2000, 2000, #EDIT THIS LINE
  end_month                           = 01,   01,   01,   #EDIT THIS LINE
  end_day                             = 13,   25,   25,   #EDIT THIS LINE
  end_hour                            = 21,   12,   12,   #EDIT THIS LINE
  interval_seconds                    = 10800             #EDIT THIS LINE
  input_from_file                     = .true.,.true.,.true.,
  history_interval                    = 180,  60,   60,
  frames_per_outfile                  = 1000, 1000, 1000,
  restart                             = .false.,
  restart_interval                    = 7200,
  io_form_history                     = 2
  io_form_restart                     = 2
  io_form_input                       = 2
  io_form_boundary                    = 2
  /
 
  &domains
  time_step                           = 180,
  time_step_fract_num                 = 0,
  time_step_fract_den                 = 1,
  max_dom                             = 1,
  use_adaptive_time_step              = .true.,
  e_we                                = 10,    112,   94,  #EDIT THIS LINE
  e_sn                                = 10,    97,    91,  #EDIT THIS LINE
  e_vert                              = 33,    33,    33,
  p_top_requested                     = 5000,
  num_metgrid_levels                  = 34,                #EDIT THIS LINE
  num_metgrid_soil_levels             = 4,
  dx                                  = 10000, 10000,  3333.33,  #EDIT THIS LINE
  dy                                  = 10000, 10000,  3333.33,  #EDIT THIS LINE
  grid_id                             = 1,     2,     3,
  parent_id                           = 0,     1,     2,
  i_parent_start                      = 1,     31,    30,
  j_parent_start                      = 1,     17,    30,
  parent_grid_ratio                   = 1,     3,     3,
  parent_time_step_ratio              = 1,     3,     3,
  feedback                            = 1,
  smooth_option                       = 0
  /

To run WRF, we will use the wrfmpi.qs file. This job script is based on the script template found in /opt/shared/templates/slurm/generic/mpi/openmpi/openmpi.qs. The openmpi.qs job script file is updated regularly, so it is suggested that you also check and compare the contents of the openmpi.qs files to see if there has been any major changes to it. If there has been, then you might want to consider adding those changes to the wrfmpi.qs job script. To run WRF, you will use sbatch to submit the job to Slurm.

  [(it_css:traine)@login01 wrf_job]$ sbatch wrfmpi.qs

After a couples minutes this script should finish running. Once the job has completed you can check the slurm-«JOB_ID».out file to check if the run was successful. If it was a successful run, then the last couple lines of the file should say something similar to the lines shown below

  Timing for main (dt= 80.00): time 2021-01-07_11:58:40 on domain   1:    0.00617 elapsed seconds
  Timing for main (dt= 80.00): time 2021-01-07_12:00:00 on domain   1:    0.00617 elapsed seconds
  Timing for Writing wrfout_d01_2021-01-07_12:00:00 for domain        1:    0.01661 elapsed seconds
  d01 2021-01-07_12:00:00 wrf: SUCCESS COMPLETE WRF
  Finished Running mpi_wrf.exe
  • software/wrf/caviness.txt
  • Last modified: 2024-04-01 15:06
  • by anita