Differences

This shows you the differences between two versions of the page.

--- software:r:caviness [2020-12-01 09:34] – frey
+++ software:r:caviness [2023-11-28 17:37] (current) – [personal/program specific R libraries and extensions] anita
@@ Line 107: / Line 107: @@
 versions than provided through VALET, or modules not available via VALET.
-R looks in an environment variable called 'R_LIBS' to obtain a list of
+R looks in an environment variable called ''R_LIBS'' to obtain a list of
 locations to search for modules.  You should ensure your entry is first
 in the list, this will allow your library to override any conflicts which
@@ Line 114: / Line 114: @@
 === Simple example ===
-Once this is done, you can install by using ''install.packages''.  Make sure you are in your workgroup (e.g. ''workgroup -g <<//investing-entity//>>''. Here
+Once this is done, you can install by using ''install.packages''.  Make sure you are in your workgroup (e.g. ''workgroup -g <<//investing-entity//>>''.  Here
 is an example:
@@ Line 125: / Line 125: @@
 $ echo $R_LIBS
 /opt/shared/r/add-ons/r3.5.1/cran/20180715
-$ R_LIBS="$WORKDIR/sw/r/add-ons/r3.5.1/testing/default:$R_LIBS"
+$ export R_LIBS="$WORKDIR/sw/r/add-ons/r3.5.1/testing/default:$R_LIBS"
 $ R -q --no-save
 > .libPaths()
@@ Line 211: / Line 211: @@
 </code>
-Notice that the output of ''.libPaths()'' specifies my personal library
+Notice that the output of ''.libPaths()'' specifies my personal library directory first?  Also it is very important to make sure all dependencies for a library are installed by using the option ''dependencies=TRUE'' for ''install.packages''.  It may be necessary to install a dependency first before trying to install the library you want if the developers of a particular library did not provide the dependency as part of the install.  In the complex example below, even though ''terra'' doesn't indicate a dependency for ''quantregForest'' it may need to be installed using ''install.packages("quantregForest", dependencies=TRUE)'' first before doing ''install.packages("terra", dependencies=TRUE)''.  Unfortunately much of this requires reviewing the developer documentation for each library to determine system requirements, dependencies and noting these dependencies may not be available or automatically installed. On top of that some dependencies require other packages to be installed first, so it can become a catch 22 situation and require trial and error to find the correct order such as needing to install ''bslib'' which is needed for ''rmarkdown'' which is needed for ''quantregForest'' installation. Lastly, make sure you look at what VALET packages are already available for installing R libraries such as the version of the compiler necessary and other system libraries like ''gdal'', ''proj'', ''geos'', etc. Using ''vpkg_devrequire'' will load other dependencies as well set environment variables such as ''LDFLAGS'', ''CPPFLAGS'', ''LIBRARY_PATH'' and ''LD_LIBRARY_PATH'' as well as ''PREFIX'' variables that are necessary in order to find the correct files during the installation of a particular R library.  For example, by loading ''gdal'' and ''geos'' we get a number of other dependencies loaded in our environment as well the appropriate environment variables that will be needed by R to install a library. As part of this trial and error, ''udunits'' failed which required quitting from R, then using ''vpkg_devrequire udunits/2.2'' to load into the environment, setting the environment variables ''UDUNITS2_INCLUDE'' and ''UDUNITS2_LIBS'' based on the environment variable ''UDUNITS_PREFIX'' by VALET, and getting back in to R to try the install again.  This is basically the trial and error process involved. Finally you will need to load all the same packages in VALET using ''vpkg_require'' and set any environment variables before trying to use the libraries you installed in your ''R_LIBS'' for this session.
-directory first?
+=== Complex example ===
+<code>
+[traine@login01.caviness ~]$ workgroup -g it_css
+[(it_css:traine)@login01.caviness ~]$ vpkg_devrequire r/4.1 gdal/3.4.3 geos/3.9.1
+Adding dependency `binutils/2.35` to your environment
+Adding dependency `gcc/11.2.0` to your environment
+Adding dependency `atlas/3.10.3` to your environment
+Adding package `r/4.1.3` to your environment
+Adding dependency `szip/2.1.1` to your environment
+Adding dependency `hdf4/4.2.13` to your environment
+Adding dependency `hdf5/1.10.2` to your environment
+Adding dependency `netcdf/4.6.1` to your environment
+Adding dependency `sqlite3/3.34.1` to your environment
+Adding dependency `proj/8.2.1` to your environment
+Adding package `gdal/3.4.3` to your environment
+Adding package `geos/3.9.1` to your environment
+[(it_css:traine)@login01.caviness ~]$ echo $LDFLAGS
+-L/opt/shared/binutils/2.35/lib -L/opt/shared/gcc/11.2.0/lib -L/opt/shared/gcc/11.2.0/lib64 -L/opt/shared/atlas/30.3/lib -L/opt/shared/r/4.1.3/lib64 -L/opt/shared/r/4.1.3/lib64/R/lib -L/opt/shared/r/4.1.3/lib64/R/lib/atlas -L/t/shared/szip/2.1.1/lib -L/opt/shared/hdf4/4.2.13/lib -L/opt/shared/hdf5/1.10.2/lib -L/opt/shared/netcdf/4.6.1/li-L/opt/shared/sqlite3/3.34.1/lib -L/opt/shared/proj/8.2.1/lib -L/opt/shared/gdal/3.4.3/lib -L/opt/shared/geos/3.9/lib
+[(it_css:traine)@login01.caviness ~]$ echo $CPPFLAGS
+-I/opt/shared/binutils/2.35/include -I/opt/shared/gcc/11.2.0/include -I/opt/shared/atlas/3.10.3/include -I/opt/shared/r/4.1.3/include -I/opt/shared/r/4.1.3/lib64/R/include -I/opt/shared/szip/2.1.1/include -I/opt/shared/hdf4/4.2.13/include -I/opt/shared/hdf5/1.10.2/include -I/opt/shared/netcdf/4.6.1/include -I/opt/shared/sqlite3/3.34.1/include -I/opt/shared/proj/8.2.1/include -I/opt/shared/gdal/3.4.3/include -I/opt/shared/geos/3.9.1/include
+[(it_css:traine)@login01.caviness ~]$ echo $LIBRARY_PATH
+/opt/shared/geos/3.9.1/lib:/opt/shared/gdal/3.4.3/lib:/opt/shared/proj/8.2.1/lib:/opt/shared/sqlite3/3.34.1/lib:/opt/shared/netcdf/4.6.1/lib:/opt/shared/hdf5/1.10.2/lib:/opt/shared/hdf4/4.2.13/lib:/opt/shared/szip/2.1.1/lib:/opt/shared/r/4.1.3/lib64/R/lib/atlas:/opt/shared/r/4.1.3/lib64/R/lib:/opt/shared/r/4.1.3/lib64:/opt/shared/atlas/3.10.3/lib:/opt/shared/gcc/11.2.0/lib64:/opt/shared/gcc/11.2.0/lib:/opt/shared/binutils/2.35/lib
+[(it_css:traine)@login01.caviness ~]$ echo $LD_LIBRARY_PATH
+/opt/shared/geos/3.9.1/lib:/opt/shared/gdal/3.4.3/lib:/opt/shared/proj/8.2.1/lib:/opt/shared/sqlite3/3.34.1/lib:/opt/shared/netcdf/4.6.1/lib:/opt/shared/hdf5/1.10.2/lib:/opt/shared/hdf4/4.2.13/lib:/opt/shared/szip/2.1.1/lib:/opt/shared/r/4.1.3/lib64/R/lib/atlas:/opt/shared/r/4.1.3/lib64/R/lib:/opt/shared/r/4.1.3/lib64:/opt/shared/atlas/3.10.3/lib:/opt/shared/gcc/11.2.0/lib64:/opt/shared/gcc/11.2.0/lib:/opt/shared/binutils/2.35/lib:/opt/shared/slurm/lib
+[(it_css:traine)@login01.caviness ~]$ env | grep PREFIX
+R_PREFIX=/opt/shared/r/4.1.3
+GDAL_PREFIX=/opt/shared/gdal/3.4.3
+GCC_PREFIX=/opt/shared/gcc/11.2.0
+GEOS_PREFIX=/opt/shared/geos/3.9.1
+HDF5_PREFIX=/opt/shared/hdf5/1.10.2
+SQLITE3_PREFIX=/opt/shared/sqlite3/3.34.1
+PROJ_PREFIX=/opt/shared/proj/8.2.1
+NETCDF_PREFIX=/opt/shared/netcdf/4.6.1
+GQUEUE_PREFIX=/opt/shared/gqueue
+HDF4_PREFIX=/opt/shared/hdf4/4.2.13
+BINUTILS_PREFIX=/opt/shared/binutils/2.35
+SZIP_PREFIX=/opt/shared/szip/2.1.1
+ATLAS_PREFIX=/opt/shared/atlas/3.10.3
+[(it_css:traine)@login01.caviness ~]$ mkdir -p $WORKDIR/sw/r/add-ons/r4.1.3/terra
+[(it_css:traine)@login01.caviness ~]$ export R_LIBS="$WORKDIR/sw/r/add-ons/r4.1.3/terra"
+[(it_css:traine)@login01.caviness ~]$ R -q --no-save
+> .libPaths()
+[1] "/work/it_css/sw/r/add-ons/r4.1.3/terra"
+[2] "/opt/shared/r/4.1.3/lib64/R/library"
+> chooseCRANmirror(all)
+Secure CRAN mirrors
+: 0-Cloud [https]
+: Australia (Canberra) [https]
+: Australia (Melbourne 1) [https]
+: Australia (Melbourne 2) [https]
+: Australia (Perth) [https]
+: Austria [https]
+ ...
+: USA (IA) [https]
+: USA (MI) [https]
+: USA (MO) [https]
+: USA (OH) [https]
+: USA (OR) [https]
+: USA (TN) [https]
+: United Arab Emirates [https]
+: Uruguay [https]
+: (other mirrors)
+Selection: 69
+> install.packages("quantregForest", dependencies=TRUE)
+Installing package into ‘/work/it_css/sw/r/add-ons/r4.1.3/terra’
+(as ‘lib’ is unspecified)
+also installing the dependencies ‘fs’, ‘R6’, ‘rappdirs’, ‘base64enc’, ‘cachem’, ‘lifecycle’, ‘memoise’, ‘mime’, ‘rlang’, ‘sass’, ‘digest’, ‘ellipsis’, ‘fastmap’, ‘cli’, ‘glue’, ‘magrittr’, ‘stringi’, ‘vctrs’, ‘evaluate’, ‘highr’, ‘xfun’, ‘yaml’, ‘bslib’, ‘fontawesome’, ‘htmltools’, ‘jquerylib’, ‘jsonlite’, ‘stringr’, ‘tinytex’, ‘randomForest’, ‘RColorBrewer’, ‘gss’, ‘knitr’, ‘rmarkdown’
+trying URL 'https://mirror.las.iastate.edu/CRAN/src/contrib/fs_1.6.3.tar.gz'
+Content type 'application/x-gzip' length 1185603 bytes (1.1 MB)
+==================================================
+downloaded 1.1 MB
+...
+...
+...
+ERROR: dependency ‘bslib’ is not available for package ‘rmarkdown’
+* removing ‘/work/it_css/sw/r/add-ons/r4.1.3/terra/rmarkdown’
+The downloaded source packages are in
+        ‘/tmp/RtmpaBulVZ/downloaded_packages’
+Warning message:
+In install.packages("quantregForest", dependencies = TRUE) :
+  installation of package ‘rmarkdown’ had non-zero exit status
+> library (quantregForest)
+Loading required package: randomForest
+randomForest 4.7-1.1
+Type rfNews() to see new features/changes/bug fixes.
+Loading required package: RColorBrewer
+> install.packages("bslib", dependencies=TRUE)
+Installing package into ‘/work/it_css/sw/r/add-ons/r4.1.3/terra’
+(as ‘lib’ is unspecified)
+also installing the dependencies ‘colorspace’, ‘utf8’, ‘prettyunits’, ‘labeling’, ‘munsell’, ‘viridisLite’, ‘fansi’, ‘pillar’, ‘pkgconfig’, ‘Rcpp’, ‘rprojroot’, ‘pkgbuild’, ‘diffobj’, ‘rematch2’, ‘gtable’, ‘isoband’, ‘scales’, ‘tibble’, ‘httpuv’, ‘xtable’, ‘sourcetools’, ‘later’, ‘promises’, ‘crayon’, ‘commonmark’, ‘brio’, ‘callr’, ‘desc’, ‘pkgload’, ‘praise’, ‘processx’, ‘ps’, ‘waldo’, ‘farver’, ‘rstudioapi’, ‘bsicons’, ‘curl’, ‘ggplot2’, ‘rmarkdown’, ‘shiny’, ‘testthat’, ‘thematic’, ‘withr’
+...
+...
+checking for udunits2.h... no
+checking for udunits2/udunits2.h... no
+checking for ut_read_xml in -ludunits2... yes
+configure: error: in `/tmp/Rtmpc5hPQz/R.INSTALL540e8c2492d/units':
+configure: error:
+--------------------------------------------------------------------------------
+  Configuration failed because udunits2.h was not found. Try installing:
+    * deb: libudunits2-dev (Debian, Ubuntu, ...)
+    * rpm: udunits2-devel (Fedora, EPEL, ...)
+    * brew: udunits (OSX)
+  If udunits2 is already installed in a non-standard location, use:
+    --configure-args='--with-udunits2-lib=/usr/local/lib'
+  if the library was not found, and/or:
+    --configure-args='--with-udunits2-include=/usr/include/udunits2'
+  if the header was not found, replacing paths with appropriate values.
+  You can alternatively set UDUNITS2_INCLUDE and UDUNITS2_LIBS manually.
+--------------------------------------------------------------------------------
+See `config.log' for more details
+ERROR: configuration failed for package ‘units’
+* removing ‘/work/it_css/sw/r/add-ons/r4.1.3/terra/units’
+The downloaded source packages are in
+        ‘/tmp/Rtmps25MZ6/downloaded_packages’
+Warning message:
+In install.packages("units", dependencies = TRUE) :
+  installation of package ‘units’ had non-zero exit status
+> quit()
+[(it_css:traine)@login01.caviness ~]$ vpkg_devrequire udunits/2.2
+Adding package `udunits/2.2.26` to your environment
+[(it_css:traine)@login01.caviness ~]$ env | grep UDUNITS
+UDUNITS_PREFIX=/opt/shared/udunits/2.2.26
+[(it_css:traine)@login01.caviness ~]$ export UDUNITS2_INCLUDE=$UDUNITS_PREFIX/include
+[(it_css:traine)@login01.caviness ~]$ export UDUNITS2_LIBS=$UDUNITS_PREFIX/lib
+[(it_css:traine)@login01.caviness ~]$ R -q --no-save
+> .libPaths()
+[1] "/work/it_css/sw/r/add-ons/r4.1.3/terra"
+[2] "/opt/shared/r/4.1.3/lib64/R/library"
+> chooseCRANmirror(all)
+Secure CRAN mirrors
+: 0-Cloud [https]
+: Australia (Canberra) [https]
+: Australia (Melbourne 1) [https]
+: Australia (Melbourne 2) [https]
+: Australia (Perth) [https]
+: Austria [https]
+...
+...
+: USA (IA) [https]
+: USA (MI) [https]
+: USA (MO) [https]
+: USA (OH) [https]
+: USA (OR) [https]
+: USA (TN) [https]
+: United Arab Emirates [https]
+: Uruguay [https]
+: (other mirrors)
+Selection: 69
+> install.packages("units", dependencies=TRUE)
+Installing package into ‘/work/it_css/sw/r/add-ons/r4.1.3/terra’
+(as ‘lib’ is unspecified)
+trying URL 'https://mirror.las.iastate.edu/CRAN/src/contrib/units_0.8-4.tar.gz'
+Content type 'application/x-gzip' length 248024 bytes (242 KB)
+==================================================
+downloaded 242 KB
+* installing *source* package ‘units’ ...
+** package ‘units’ successfully unpacked and MD5 sums checked
+** using staged installation
+configure: units: 0.8-4
+checking whether the C++ compiler works... yes
+checking for C++ compiler default output file name... a.out
+checking for suffix of executables...
+checking whether we are cross compiling... no
+checking for suffix of object files... o
+checking whether the compiler supports GNU C++... yes
+checking whether /opt/shared/gcc/11.2.0/bin/g++ -std=gnu++14 accepts -g... yes
+checking for /opt/shared/gcc/11.2.0/bin/g++ -std=gnu++14 option to enable C++11 features... none needed
+checking for stdio.h... yes
+checking for stdlib.h... yes
+checking for string.h... yes
+checking for inttypes.h... yes
+checking for stdint.h... yes
+checking for strings.h... yes
+checking for sys/stat.h... yes
+checking for sys/types.h... yes
+checking for unistd.h... yes
+checking for _Bool... no
+checking for stdbool.h that conforms to C99... yes
+checking for error_at_line... yes
+checking for gcc... /opt/shared/gcc/11.2.0/bin/gcc
+checking whether the compiler supports GNU C... yes
+checking whether /opt/shared/gcc/11.2.0/bin/gcc accepts -g... yes
+checking for /opt/shared/gcc/11.2.0/bin/gcc option to enable C11 features... none needed
+checking for XML_ParserCreate in -lexpat... yes
+checking for udunits2.h... yes
+checking for ut_read_xml in -ludunits2... yes
+configure: creating ./config.status
+config.status: creating src/Makevars
+** libs
+/opt/shared/gcc/11.2.0/bin/g++ -std=gnu++14 -I"/opt/shared/r/4.1.3/lib64/R/include" -DNDEBUG -DUDUNITS2_DIR=0 -I/opt/shared/udunits/2.2.26/include -I/opt/shared/udunits/2.2.26/include -I/opt/shared/binutils/2.35/include -I/opt/shared/gcc/11.2.0/include -I/opt/shared/r/4.1.3/include -I'/work/it_css/sw/r/add-ons/r4.1.3/terra/Rcpp/include' -I/opt/shared/binutils/2.35/include -I/opt/shared/gcc/11.2.0/include -I/opt/shared/r/4.1.3/include   -fpic  -g -O2  -c RcppExports.cpp -o RcppExports.o
+/opt/shared/gcc/11.2.0/bin/g++ -std=gnu++14 -I"/opt/shared/r/4.1.3/lib64/R/include" -DNDEBUG -DUDUNITS2_DIR=0 -I/opt/shared/udunits/2.2.26/include -I/opt/shared/udunits/2.2.26/include -I/opt/shared/binutils/2.35/include -I/opt/shared/gcc/11.2.0/include -I/opt/shared/r/4.1.3/include -I'/work/it_css/sw/r/add-ons/r4.1.3/terra/Rcpp/include' -I/opt/shared/binutils/2.35/include -I/opt/shared/gcc/11.2.0/include -I/opt/shared/r/4.1.3/include   -fpic  -g -O2  -c udunits.cpp -o udunits.o
+/opt/shared/gcc/11.2.0/bin/g++ -std=gnu++14 -shared -L/opt/shared/r/4.1.3/lib64/R/lib -L/opt/shared/binutils/2.35/lib -L/opt/shared/gcc/11.2.0/lib -L/opt/shared/gcc/11.2.0/lib64 -I/opt/shared/r/4.1.3/lib64 -o units.so RcppExports.o udunits.o -lexpat -L/opt/shared/udunits/2.2.26/lib -lexpat -ludunits2 -L/opt/shared/r/4.1.3/lib64/R/lib -lR
+installing to /work/it_css/sw/r/add-ons/r4.1.3/terra/00LOCK-units/00new/units/libs
+** R
+** demo
+** inst
+** byte-compile and prepare package for lazy loading
+** help
+*** installing help indices
+** building package indices
+** installing vignettes
+** testing if installed package can be loaded from temporary location
+** checking absolute paths in shared objects and dynamic libraries
+** testing if installed package can be loaded from final location
+** testing if installed package keeps a record of temporary installation path
+* DONE (units)
+> library(terra)
+terra 1.7.55
+> library(quantregForest)
+Loading required package: randomForest
+randomForest 4.7-1.1
+Type rfNews() to see new features/changes/bug fixes.
+Loading required package: RColorBrewer
+> quit()
+[(it_css:traine)@login01.caviness ~]$
+</code>
 === Using IT's udbuild environment ===
@@ Line 296: / Line 520: @@
 ==== matmul.qs file ====
-To run a R script in batch instead of on the command line has nearly the same steps. Copy a template job submission script (''/opt/templates/slurm/generic/threads.qs'') for example and called it ''matmul.qs''.  Now edit it to change the job name and add your commands for your job something like this:
+To run a R script in batch instead of on the command line has nearly the same steps. Copy a template job submission script (''/opt/shared/templates/slurm/generic/threads.qs'') for example and call it ''matmul.qs''.  Now edit it to change the job name and add your commands for your job something like this:
-<code bash>
+<file bash matmul.qs>
 #!/bin/bash -l
 #
@@ Line 378: / Line 602: @@
 ===== sweep.qs file =====
-Again copy a template job submission script (/opt/templates/slurm/generic/threads.qs) for example and call it ''sweep.qs''. Now edit it to change the job name, this time adding options for an array job and add your commands for your job something like this:
+Again copy a template job submission script (/opt/shared/templates/slurm/generic/threads.qs) for example and call it ''sweep.qs''. Now edit it to change the job name, this time adding options for an array job and add your commands for your job something like this:
 <file bash sweep.qs>