Differences

This shows you the differences between two versions of the page.

--- technical:whitepaper:r-runtime-blas-lapack [2018-12-10 11:54] – [Substituting an alternate library] frey
+++ technical:whitepaper:r-runtime-blas-lapack [2018-12-10 12:40] (current) – [R: Runtime-configuration BLAS/LAPACK] frey
@@ Line 8: / Line 8: @@
     * can require on the order of //N// times the disk space of a single build
     * puts a greater burden on the sysadmin to maintain all //N// similarly-outfitted copies
-  - R only makes use of standardized BLAS/LAPACK APIs, so any standard BLAS/LAPACK library should be able to be chosen at runtime (not just build time)/
+  - R only makes use of standardized BLAS/LAPACK APIs, so any standard BLAS/LAPACK library should be able to be chosen at runtime (not just build time).
 ===== Substituting an alternate library =====
@@ Line 16: / Line 16: @@
   * [[https://stackoverflow.com/questions/29984141/does-installing-blas-atlas-mkl-openblas-will-speed-up-r-package-that-is-written]]
 The basic idea is:
-  * copy ''libatlas.so'' to ''**R_PREFIX**/lib64/R/lib''
+  * copy ''libatlas.so'' to ''** R_HOME**/lib64/R/lib''
-  * remove ''libRblas.so'' and ''libRlapack.so'' from ''**R_PREFIX**/lib64/R/lib''
+  * remove ''libRblas.so'' and ''libRlapack.so'' from ''** R_HOME**/lib64/R/lib''
-  * symlink ''libRblas.so'' and ''libRlapack.so'' to ''libatlas.so'' in ''**R_PREFIX**/lib64/R/lib''
+  * symlink ''libRblas.so'' and ''libRlapack.so'' to ''libatlas.so'' in ''** R_HOME**/lib64/R/lib''
-This copy of R is configured to use ''**R_PREFIX**/lib64/R/lib'' to resolve shared libraries, so when executing the ''R'' command, for example, the symlinks will lead the runtime linker to the ATLAS library when resolving BLAS/LAPACK functions.
+This copy of R is configured to use ''** R_HOME**/lib64/R/lib'' to resolve shared libraries, so when executing the ''R'' command, for example, the symlinks will lead the runtime linker to the ATLAS library when resolving BLAS/LAPACK functions.
 This scheme requires two things:
@@ Line 31: / Line 31: @@
 ^Path  ^Description^
-|''**R_PREFIX**/lib64/R/lib''  |base directory where R looks for shared libraries by default|
+|''** R_HOME**/lib64/R/lib''  |base directory where R looks for shared libraries by default|
-|''**R_PREFIX**/lib64/R/lib/libRblas.so''  |symlink to chosen BLAS library (from one of the subdirectories herein)|
+|''** R_HOME**/lib64/R/lib/libRblas.so''  |symlink to chosen BLAS library (from one of the subdirectories herein)|
-|''**R_PREFIX**/lib64/R/lib/libRlapack.so''  |symlink to chosen LAPACK library (from one of the subdirectories herein)|
+|''** R_HOME**/lib64/R/lib/libRlapack.so''  |symlink to chosen LAPACK library (from one of the subdirectories herein)|
-|''**R_PREFIX**/lib64/R/lib/atlas''  |directory to hold ''libatlas.so''|
+|''** R_HOME**/lib64/R/lib/atlas''  |directory to hold ''libatlas.so''|
-|''**R_PREFIX**/lib64/R/lib/rblas''  |directory to hold the bundled ''libRblas.so'' and ''libRlapack.so'' produced by R build procedure|
+|''** R_HOME**/lib64/R/lib/rblas''  |directory to hold the bundled ''libRblas.so'' and ''libRlapack.so'' produced by R build procedure|
-|''**R_PREFIX**/lib64/R/lib/mkl''  |directory to hold MKL variants|
+|''** R_HOME**/lib64/R/lib/mkl''  |directory to hold MKL variants|
-|''**R_PREFIX**/lib64/R/lib/mkl/seq''  |directory to hold sequential MKL variant|
+|''** R_HOME**/lib64/R/lib/mkl/seq''  |directory to hold sequential MKL variant|
-|''**R_PREFIX**/lib64/R/lib/mkl/thr''  |directory to hold threaded MKL variant|
+|''** R_HOME**/lib64/R/lib/mkl/thr''  |directory to hold threaded MKL variant|
-The ATLAS library contains both BLAS and LAPACK APIs in a single shared library and both the ''libRblas.so'' and 'libRatlas.so'' symlinks are pointed to it.  The Intel MKL contains both APIs, as well, but is modularized by the parallel nature of the runtime environment:  sequential (non-threaded) or OpenMP (multithreaded).  Our solution is to build a shim library linked to the appropriate Intel libraries.
+==== R BLAS/LAPACK ====
-==== Sequential MKL shim ====
+When we restructured the R ''lib64/R/lib'' directory, the bundled ''libRblas.so'' and ''libRlapack.so'' shared library files were moved to the ''rblas'' subdirectory.  To configure R to use its bundled libraries:
-A C source file containing a dummy function was created in ''**R_PREFIX**/lib64/R/lib/mkl/seq'':
+<code bash>
+$ cd ${R_HOME}/lib64/R/lib
+$ rm -f libR{blas,lapack}.so
+$ ln -s rblas/libRblas.so .
+$ ln -s rblas/libRlapack.so .
+</code>
+==== ATLAS ====
+The ATLAS library contains both BLAS and LAPACK APIs in a single shared library.  With ''libatlas.so'' copied into the ''** R_HOME**/lib64/R/lib/atlas'' subdirectory, we configure R to use ATLAS:
+<code bash>
+$ cd ${R_HOME}/lib64/R/lib
+$ rm -f libR{blas,lapack}.so
+$ ln -s atlas/libRblas.so .
+$ ln -s atlas/libRlapack.so .
+</code>
+==== Sequential MKL ====
+A C source file containing a dummy function was created in ''** R_HOME**/lib64/R/lib/mkl/seq'':
 <file C shim.c>
@@ Line 57: / Line 77: @@
 <code bash>
-$ cd ${R_PREFIX}/lib64/R/lib/mkl/seq
+$ cd ${R_HOME}/lib64/R/lib/mkl/seq
 $ icc -shared -o libRblas.so -mkl=sequential shim.c
 $ ln -s libRblas.so libRlapack.so
 </code>
-==== Threaded MKL shim ====
+To configure R to use the sequential MKL:
-A C source file containing a dummy function was created in ''**R_PREFIX**/lib64/R/lib/mkl/thr'':
+<code bash>
+$ cd ${R_HOME}/lib64/R/lib
+$ rm -f libR{blas,lapack}.so
+$ ln -s mkl/seq/libRblas.so .
+$ ln -s mkl/seq/libRlapack.so .
+</code>
+==== Threaded MKL ====
+A C source file containing a dummy function was created in ''** R_HOME**/lib64/R/lib/mkl/thr'':
 <file C shim.c>
@@ Line 77: / Line 106: @@
 <code bash>
-$ cd ${R_PREFIX}/lib64/R/lib/mkl/thr
+$ cd ${R_HOME}/lib64/R/lib/mkl/thr
 $ icc -shared -o libRblas.so shim.c -lmkl_gnu_thread -lmkl_core -lmkl_intel_lp64
 $ ln -s libRblas.so libRlapack.so
+</code>
+To configure R to use the threaded MKL:
+<code bash>
+$ cd ${R_HOME}/lib64/R/lib
+$ rm -f libR{blas,lapack}.so
+$ ln -s mkl/thr/libRblas.so .
+$ ln -s mkl/thr/libRlapack.so .
 </code>
 ===== Runtime-configurable substitution =====
+By stashing each BLAS/LAPACK variant in its own subdirectory, our copy of R is actually fairly close to being runtime-configurable with respect to choice of BLAS/LAPACK.  Since all R commands will setup the environment to have the runtime linker check ''** R_HOME**/lib64/R/lib'' for shared libraries, the ''libRblas.so'' and ''libRlapack.so'' symlinks in that directory will always have priority over any other path we might add to ''LD_LIBRARY_PATH'' prior to issuing the ''R'' command, for example.  However, if ''libRblas.so'' and ''libRlapack.so'' are **not** present in that directory, the runtime linker will be forced to consult other paths present in ''LD_LIBRARY_PATH''.
+On our Caviness cluster we include no BLAS/LAPACK library symlinks in the base directory which R checks for shared libraries:
+<code bash>
+$ cd ${R_HOME}/lib64/R/lib
+$ rm -f libR{blas,lapack}.so
+</code>
+In our VALET package definition for R we configure four variants that differ by BLAS/LAPACK (as a feature tag)
+<code bash>
+$ vpkg_versions r
+   :
+r                The R Project for Statistical Computing
+.5            alias to r/3.5.1
+* 3.5.1          R 3.5.1 with system compilers, ATLAS
+.5.1:mkl-seq  R 3.5.1 with system compilers, MKL (sequential)
+.5.1:mkl-thr  R 3.5.1 with system compilers, MKL (multithread)
+.5.1:rblas    R 3.5.1 with system compilers, R reference BLAS/LAPACK
+</code>
+with ATLAS as the default (recommended) choice of underlying BLAS/LAPACK.  Each variant of the 3.5.1 version uses the same installation prefix, but adds a unique BLAS/LAPACK subdirectory to ''LD_LIBRARY_PATH'' when added to the user's environment:
+<code bash>
+$ vpkg_require r/3.5.1
+Adding package `r/3.5.1` to your environment
+$ echo $LD_LIBRARY_PATH
+/opt/shared/r/3.5.1/lib64/R/lib/atlas: ...
+$ vpkg_rollback all
+$ vpkg_require r/3.5.1:rblas
+Adding package `r/3.5.1:rblas` to your environment
+$ echo $LD_LIBRARY_PATH
+/opt/shared/r/3.5.1/lib64/R/lib/rblas:
+</code>
+This works fine so long as you don't attempt to install any R modules that require the BLAS/LAPACK functionality.  The R module-building environment defaults to using link search paths in the base directory (''** R_HOME**/lib64/R/lib'') and fails to find ''libRblas.so'' or ''libRlapack.so''.  This is easily fixed by altering two lines in the R installation's standard make flags file at ''** R_HOME**/lib64/R/etc/Makeconf'':
+<code>
+BLAS_LIBS = -L"$(R_HOME)/lib$(R_ARCH)/$(R_BLAS_VARIANT)" -lRblas
+</code>
+and
+<code>
+LAPACK_LIBS = -L"$(R_HOME)/lib$(R_ARCH)/$(R_BLAS_VARIANT)" -lRlapack
+</code>
+The link search added here will default to the usual path if ''R_BLAS_VARIANT'' is not set in the user's environment.  But in our VALET configuration of each of the variants, we set the appropriate relative path for ''R_BLAS_VARIANT'':
+<code bash>
+$ vpkg_require r/3.5.1:rblas
+Adding package `r/3.5.1:rblas` to your environment
+$ echo $R_BLAS_VARIANT
+rblas
+$ vpkg_rollback all
+$ vpkg_require r/3.5.1:mkl-seq
+Adding package `r/3.5.1:mkl-seq` to your environment
+$ echo $R_BLAS_VARIANT
+mkl/seq
+</code>
+We now have a runtime-configurable BLAS/LAPACK for this single installation of R, and any properly-packaged R modules should build fine against it.
+===== VALET configuration =====
+Here is the VALET configuration for our installation of R 3.5.1 with runtime-configurable BLAS/LAPACK:
+<file yaml>
+r:
+  description:       The R Project for Statistical Computing
+  url:               http://www.r-project.org/
+  prefix:            /opt/shared/r
+  default-version:   "3.5.1"
+  versions:
+    "3.5":
+      alias-to:      3.5.1
+    "3.5.1":
+      description:   R 3.5.1 with system compilers, ATLAS
+      actions:
+        - libdir:    lib64/R/lib
+        - incdir:    lib64/R/include
+        - libdir:    lib64/R/lib/atlas
+        - variable:  R_BLAS_VARIANT
+          operator:  set
+          value:     atlas
+    "3.5.1:rblas":
+      description:   R 3.5.1 with system compilers, R reference BLAS/LAPACK
+      prefix:        3.5.1
+      actions:
+        - libdir:    lib64/R/lib
+        - incdir:    lib64/R/include
+        - libdir:    lib64/R/lib/rblas
+        - variable:  R_BLAS_VARIANT
+          operator:  set
+          value:     rblas
+    "3.5.1:mkl-seq":
+      description:   R 3.5.1 with system compilers, MKL (sequential)
+      prefix:        3.5.1
+      actions:
+        - libdir:    lib64/R/lib
+        - incdir:    lib64/R/include
+        - libdir:    lib64/R/lib/mkl/seq
+        - variable:  R_BLAS_VARIANT
+          operator:  set
+          value:     mkl/seq
+    "3.5.1:mkl-thr":
+      description:   R 3.5.1 with system compilers, MKL (multithread)
+      prefix:        3.5.1
+      actions:
+        - libdir:    lib64/R/lib
+        - incdir:    lib64/R/include
+        - libdir:    lib64/R/lib/mkl/thr
+        - variable:  R_BLAS_VARIANT
+          operator:  set
+          value:     mkl/thr
+</file>