Differences

This shows you the differences between two versions of the page.

--- technical:whitepaper:r-runtime-blas-lapack [2018-12-10 11:37] – [Substituting an alternate library] frey
+++ technical:whitepaper:r-runtime-blas-lapack [2018-12-10 12:31] – [Runtime-configurable substitution] frey
@@ Line 32: / Line 32: @@
 ^Path  ^Description^
 |''**R_PREFIX**/lib64/R/lib''  |base directory where R looks for shared libraries by default|
+|''**R_PREFIX**/lib64/R/lib/libRblas.so''  |symlink to chosen BLAS library (from one of the subdirectories herein)|
+|''**R_PREFIX**/lib64/R/lib/libRlapack.so''  |symlink to chosen LAPACK library (from one of the subdirectories herein)|
 |''**R_PREFIX**/lib64/R/lib/atlas''  |directory to hold ''libatlas.so''|
 |''**R_PREFIX**/lib64/R/lib/rblas''  |directory to hold the bundled ''libRblas.so'' and ''libRlapack.so'' produced by R build procedure|
@@ Line 38: / Line 40: @@
 |''**R_PREFIX**/lib64/R/lib/mkl/thr''  |directory to hold threaded MKL variant|
+==== R BLAS/LAPACK ====
+When we restructured the R ''lib64/R/lib'' directory, the bundled ''libRblas.so'' and ''libRlapack.so'' shared library files were moved to the ''rblas'' subdirectory.  To configure R to use its bundled libraries:
+<code bash>
+$ cd ${R_PREFIX}/lib64/R/lib
+$ rm -f libR{blas,lapack}.so
+$ ln -s rblas/libRblas.so .
+$ ln -s rblas/libRlapack.so .
+</code>
+==== ATLAS ====
+The ATLAS library contains both BLAS and LAPACK APIs in a single shared library.  With ''libatlas.so'' copied into the ''**R_PREFIX**/lib64/R/lib/atlas'' subdirectory, we configure R to use ATLAS:
+<code bash>
+$ cd ${R_PREFIX}/lib64/R/lib
+$ rm -f libR{blas,lapack}.so
+$ ln -s atlas/libRblas.so .
+$ ln -s atlas/libRlapack.so .
+</code>
+==== Sequential MKL ====
+A C source file containing a dummy function was created in ''**R_PREFIX**/lib64/R/lib/mkl/seq'':
+<file C shim.c>
+int
+mkl_shim_dummy(void)
+{
+	return 0;
+}
+</file>
+The shim library is then created thusly:
+<code bash>
+$ cd ${R_PREFIX}/lib64/R/lib/mkl/seq
+$ icc -shared -o libRblas.so -mkl=sequential shim.c
+$ ln -s libRblas.so libRlapack.so
+</code>
+To configure R to use the sequential MKL:
+<code bash>
+$ cd ${R_PREFIX}/lib64/R/lib
+$ rm -f libR{blas,lapack}.so
+$ ln -s mkl/seq/libRblas.so .
+$ ln -s mkl/seq/libRlapack.so .
+</code>
+==== Threaded MKL ====
+A C source file containing a dummy function was created in ''**R_PREFIX**/lib64/R/lib/mkl/thr'':
+<file C shim.c>
+int
+mkl_shim_dummy(void)
+{
+	return 0;
+}
+</file>
+Since our R build used the GNU C compiler, the threaded MKL variant only works if the shim library is built against the GNU OpenMP runtime.  Using just "-mkl=parallel" links against the Intel OpenMP runtime which in testing yielded numerical issues (not actual crashes).  The shim library is then created thusly:
+<code bash>
+$ cd ${R_PREFIX}/lib64/R/lib/mkl/thr
+$ icc -shared -o libRblas.so shim.c -lmkl_gnu_thread -lmkl_core -lmkl_intel_lp64
+$ ln -s libRblas.so libRlapack.so
+</code>
+To configure R to use the threaded MKL:
+<code bash>
+$ cd ${R_PREFIX}/lib64/R/lib
+$ rm -f libR{blas,lapack}.so
+$ ln -s mkl/thr/libRblas.so .
+$ ln -s mkl/thr/libRlapack.so .
+</code>
 ===== Runtime-configurable substitution =====
+By stashing each BLAS/LAPACK variant in its own subdirectory, our copy of R is actually fairly close to being runtime-configurable with respect to choice of BLAS/LAPACK.  Since all R commands will setup the environment to have the runtime linker check ''**R_PREFIX**/lib64/R/lib'' for shared libraries, the ''libRblas.so'' and ''libRlapack.so'' symlinks in that directory will always have priority over any other path we might add to ''LD_LIBRARY_PATH'' prior to issuing the ''R'' command, for example.  However, if ''libRblas.so'' and ''libRlapack.so'' are **not** present in that directory, the runtime linker will be forced to consult other paths present in ''LD_LIBRARY_PATH''.
+On our Caviness cluster we include no BLAS/LAPACK library symlinks in the base directory which R checks for shared libraries:
+<code bash>
+$ cd ${R_PREFIX}/lib64/R/lib
+$ rm -f libR{blas,lapack}.so
+</code>
+In our VALET package definition for R we configure four variants that differ by BLAS/LAPACK (as a feature tag)
+<code bash>
+$ vpkg_versions r
+   :
+r                The R Project for Statistical Computing
+.5            alias to r/3.5.1
+* 3.5.1          R 3.5.1 with system compilers, ATLAS
+.5.1:mkl-seq  R 3.5.1 with system compilers, MKL (sequential)
+.5.1:mkl-thr  R 3.5.1 with system compilers, MKL (multithread)
+.5.1:rblas    R 3.5.1 with system compilers, R reference BLAS/LAPACK
+</code>
+with ATLAS as the default (recommended) choice of underlying BLAS/LAPACK.  Each variant of the 3.5.1 version uses the same installation prefix, but adds a unique BLAS/LAPACK subdirectory to ''LD_LIBRARY_PATH'' when added to the user's environment:
+<code bash>
+$ vpkg_require r/3.5.1
+Adding package `r/3.5.1` to your environment
+$ echo $LD_LIBRARY_PATH
+/opt/shared/r/3.5.1/lib64/R/lib/atlas: ...
+$ vpkg_rollback all
+$ vpkg_require r/3.5.1:rblas
+Adding package `r/3.5.1:rblas` to your environment
+$ echo $LD_LIBRARY_PATH
+/opt/shared/r/3.5.1/lib64/R/lib/rblas:
+</code>
+This works fine so long as you don't attempt to install any R modules that require the BLAS/LAPACK functionality.  The R module-building environment defaults to using link search paths in the base directory (''**R_PREFIX**/lib64/R/lib'') and fails to find ''libRblas.so'' or ''libRlapack.so''.  This is easily fixed by altering two lines in the R installation's standard make flags file at ''**R_PREFIX**/lib64/R/etc/Makeconf'':
+<code>
+BLAS_LIBS = -L"$(R_HOME)/lib$(R_ARCH)/$(R_BLAS_VARIANT)" -lRblas
+</code>
+and
+<code>
+LAPACK_LIBS = -L"$(R_HOME)/lib$(R_ARCH)/$(R_BLAS_VARIANT)" -lRlapack
+</code>