This is an old revision of the document!
Adding your own library of R modules in R_LIBS
The following instructions were adapted from installing personal/program specific R libraries and extensions on Caviness.
List of packages requested to be installed
- rlang - needed to install an updated version for other requested packages
- Rcpp - needed to install an updated version for other requested packages
- ranger - install first gets loaded by VSURF
- randomForest
- VSURF
- plyr
- ipred
- e1071
- quantregForest
- car
- rgdal
- rasterVis
- Raster
- units - needed to install an updated version for other requested packages
- sf - needed updated version
- reproducible - needed updated version
- SpaDES
- foreach
- doParallel
- openxlsx
- DescTools
- ggplot2
Packages in bold were not requested, but necessary in order to update the R packages requested.
There is much research required to determine what other software needs to be loaded via VALET, plus which packages need to be updated and in what order so the list of requested packages will install correctly. For this example, SpaDES, is one of the packages we want to install so use SpaDES: Develop and Run Spatially Explicit Discrete Event Simulation Models to see the dependencies and imports required. This step should be repeated for each package.
Preparations
Make sure you connect to Caviness with X11 enable (Xming for Windows, XQuartz for Mac) before starting as some of the packages need X11 to compile properly.
Next make sure you are in your workgroup (ie. workgroup -g «investing_entity»
).
Next choose a directory in which to install the R libraries. This will depend on a number of factors, but mostly the version of R. The strings «r-version»
and «rlibs-name»
will denote the version of R and the name chosen for the R libraries – this recipe will use r3.5.1
and spatial-env
. Do not locate this directory under the /lustre/scratch
file system; typically a directory under the workgroup's storage is appropriate:
- If adding the R libraries for multiple users in the workgroup, choose
${WORKDIR}/sw/r/add-ons/«r-version»/«rlibs-name»/default
as the base directory. - If the R libraries are solely for personal use, choose
${WORKDIR}/users/<username>/sw/r/add-ons/«r-version»/«rlibs-name»/default
, for example.
Note that these examples assume a standard workgroup storage layout with group-writable sw
and users
directories at the top level. Create the directory:
$ mkdir -p ${WORKDIR}/sw/r/add-ons/r3.5.1/spatial-env/default
Now load all the packages via VALET that are needed to install the R libraries and add the new path to the R_LIBS
environment variable for the installation. Due to the number of packages and environment variables required, it is best to create a script called setup-«rlibs-name».sh
. This script serves also a document for future installations to know what packages and environment variables were required for the installation.
$ cat setup-spatial-env.sh #!/bin/bash # usage: source install-spatial-env.sh # clear environment vpkg_rollback all # Load software via VALET and set environment variables # that are needed to install requested R packages: # ranger VSURF plyr randomForest ipred e1071 quantregForest # car rgdal raster rasterVis SpaDES foreach doParallel # openxlsx DescTools ggplot2 vpkg_devrequire r/3.5.1:mkl-thr vpkg_devrequire r-cran/3.5.1:20180715 vpkg_devrequire gdal/2.3.0 vpkg_devrequire proj/5.1.0 vpkg_devrequire netcdf/4.6.1 vpkg_devrequire udunits/2.2.26 export UDUNITS2_LIBS=${UDUNITS_PREFIX}/lib export UDUNITS2_INCLUDE=${UDUNITS_PREFIX}/include vpkg_devrequire geos/3.6.2 export GEOS_DIR=${GEOS_PREFIX} # Add the new R library path created to the R_LIBS environment # (i.e.) based on doing "mkdir -p ${WORKDIR}/sw/r/add-ons/r3.5.1/spatial-env/default" R_LIBS="${WORKDIR}/sw/r/add-ons/r3.5.1/spatial-env/default:${R_LIBS}" # display R_LIBS with the new R libraries directory added echo ${R_LIBS}
Install the R Libraries
In most cases, creating a R script will be necessary due to the large number of packages and information required to install each library such as install-«rlibs-name».R
. This is better suited due to the amount of typing, potential mistakes and again a way of documenting your steps for future installations. For this example we will call it install-spatial-env.R
$ more install-spatial-env.R # usage: R CMD BATCH install-spatial-env.R & .libPaths() chooseCRANmirror(ind = 56) install.packages("rlang", type = "source", dependencies=TRUE) library(rlang) install.packages("Rcpp", type = "source", dependencies=TRUE) library(Rcpp) install.packages("ranger", type = "source", configure.args=c('--with-proj-lib=/opt/shared/proj/5.1.0/lib', '--with-proj-include=/opt/shared/proj/5.1.0/include'), dependencies = TRUE) library(ranger) install.packages("randomForest", type = "source", configure.args=c('--with-proj-lib=/opt/shared/proj/5.1.0/lib', '--with-proj-include=/opt/shared/proj/5.1.0/include'), dependencies = TRUE) library(randomForest) install.packages("VSURF", type = "source", configure.args=c('--with-proj-lib=/opt/shared/proj/5.1.0/lib', '--with-proj-include=/opt/shared/proj/5.1.0/include'), dependencies = TRUE) library(VSURF) install.packages("plyr", type = "source", configure.args=c('--with-proj-lib=/opt/shared/proj/5.1.0/lib', '--with-proj-include=/opt/shared/proj/5.1.0/include'), dependencies = TRUE) library(plyr) install.packages("ipred", type = "source", configure.args=c('--with-proj-lib=/opt/shared/proj/5.1.0/lib', '--with-proj-include=/opt/shared/proj/5.1.0/include'), dependencies = TRUE) library(ipred) install.packages("e1071", type = "source", configure.args=c('--with-proj-lib=/opt/shared/proj/5.1.0/lib', '--with-proj-include=/opt/shared/proj/5.1.0/include'), dependencies = TRUE) library(e1071) install.packages("quantregForest", type = "source", configure.args=c('--with-proj-lib=/opt/shared/proj/5.1.0/lib', '--with-proj-include=/opt/shared/proj/5.1.0/include'), dependencies = TRUE) library(quantregForest) install.packages("car", type = "source", configure.args=c('--with-proj-lib=/opt/shared/proj/5.1.0/lib', '--with-proj-include=/opt/shared/proj/5.1.0/include'), dependencies = TRUE) install.packages("rgdal", type = "source", configure.args=c('--with-proj-lib=/opt/shared/proj/5.1.0/lib', '--with-proj-include=/opt/shared/proj/5.1.0/include'), dependencies = TRUE) install.packages("rasterVis", type = "source", configure.args=c('--with-proj-lib=/opt/shared/proj/5.1.0/lib', '--with-proj-include=/opt/shared/proj/5.1.0/include'), dependencies = TRUE) library(rasterVis) install.packages("raster", type = "source", configure.args=c('--with-proj-lib=/opt/shared/proj/5.1.0/lib', '--with-proj-include=/opt/shared/proj/5.1.0/include'), dependencies = TRUE) install.packages("units", type = "source", configure.args=c('--with-udunits2-lib=/opt/shared/udunits/2.2.26/lib', '--with-udunits2-include=/opt/shared/udunits/2.2.26/include'), dependencies=TRUE) library(units) install.packages("sf", dependencies = TRUE) library(sf) install.packages("reproducible", type = "source", dependencies = TRUE) library(reproducible) install.packages("SpaDES", type = "source", configure.args=c('--with-udunits2-lib=/opt/shared/udunits/2.2.26/lib', '--with-udunits2-include=/opt/shared/udunits/2.2.26/include'), dependencies=TRUE) library(SpaDES) library(foreach) library(doParallel) library(openxlsx) library(DescTools) library(ggplot2)
Now install the R libraries by running the R script on a compute node using the devel
partition,
[(it_css:traine)@login01 ~]$ salloc --partition=devel --nodes=1 --ntasks=1 --cpus-per-task=4 --time=120 salloc: Pending job allocation 9441286 salloc: job 9441286 queued and waiting for resources salloc: job 9441286 has been allocated resources salloc: Granted job allocation 9441286 salloc: Waiting for resource configuration salloc: Nodes r00n56 are ready for job [traine@r00n56 ~]$ R CMD BATCH install-spatial-env.R & [traine@r00n56 ~]$
and by using &
it will run the R script in the background since this installation may take a long time, hopefully not longer than 2 hours (120 minutes) which is the maximum time on the devel
partition. When the R script has finished, you should see something like the following at the end of the output file install-spatial-env.Rout
> proc.time() user system elapsed 779.450 84.861 1262.779
Review Install Output
Review the generated output file, install-spatial-env.Rout
, for details on each package installation. If necessary adjust the install R script and rerun. If a package was successfully downloaded and installed, it will not redo it again, it will see the package exists and will pick up where it left off.
Final List of R packages Installed
$ ls $WORKDIR/sw/r/add-ons/r3.5.1/spatial-env/default car leafem quantregForest rgdal SpaDES.tools dplyr leaflet quickPlot rgeos stars e1071 leafpop randomForest rlang tibble ellipsis leafsync ranger scales tidyselect fansi mapview raster sf units gdalUtils ncdf4 rasterVis SpaDES vctrs ggforce pillar Rcpp SpaDES.addins VSURF ipred plyr reproducible SpaDES.core