technical:whitepaper:r-runtime-blas-lapack

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
technical:whitepaper:r-runtime-blas-lapack [2018-12-10 11:38] – [Substituting an alternate library] freytechnical:whitepaper:r-runtime-blas-lapack [2018-12-10 12:40] (current) – [R: Runtime-configuration BLAS/LAPACK] frey
Line 8: Line 8:
     * can require on the order of //N// times the disk space of a single build     * can require on the order of //N// times the disk space of a single build
     * puts a greater burden on the sysadmin to maintain all //N// similarly-outfitted copies     * puts a greater burden on the sysadmin to maintain all //N// similarly-outfitted copies
-  - R only makes use of standardized BLAS/LAPACK APIs, so any standard BLAS/LAPACK library should be able to be chosen at runtime (not just build time)/+  - R only makes use of standardized BLAS/LAPACK APIs, so any standard BLAS/LAPACK library should be able to be chosen at runtime (not just build time).
  
 ===== Substituting an alternate library ===== ===== Substituting an alternate library =====
Line 16: Line 16:
   * [[https://stackoverflow.com/questions/29984141/does-installing-blas-atlas-mkl-openblas-will-speed-up-r-package-that-is-written]]   * [[https://stackoverflow.com/questions/29984141/does-installing-blas-atlas-mkl-openblas-will-speed-up-r-package-that-is-written]]
 The basic idea is: The basic idea is:
-  * copy ''libatlas.so'' to ''**R_PREFIX**/lib64/R/lib'' +  * copy ''libatlas.so'' to ''** R_HOME**/lib64/R/lib'' 
-  * remove ''libRblas.so'' and ''libRlapack.so'' from ''**R_PREFIX**/lib64/R/lib'' +  * remove ''libRblas.so'' and ''libRlapack.so'' from ''** R_HOME**/lib64/R/lib'' 
-  * symlink ''libRblas.so'' and ''libRlapack.so'' to ''libatlas.so'' in ''**R_PREFIX**/lib64/R/lib'' +  * symlink ''libRblas.so'' and ''libRlapack.so'' to ''libatlas.so'' in ''** R_HOME**/lib64/R/lib'' 
-This copy of R is configured to use ''**R_PREFIX**/lib64/R/lib'' to resolve shared libraries, so when executing the ''R'' command, for example, the symlinks will lead the runtime linker to the ATLAS library when resolving BLAS/LAPACK functions.+This copy of R is configured to use ''** R_HOME**/lib64/R/lib'' to resolve shared libraries, so when executing the ''R'' command, for example, the symlinks will lead the runtime linker to the ATLAS library when resolving BLAS/LAPACK functions.
  
 This scheme requires two things: This scheme requires two things:
Line 31: Line 31:
  
 ^Path  ^Description^ ^Path  ^Description^
-|''**R_PREFIX**/lib64/R/lib''  |base directory where R looks for shared libraries by default| +|''** R_HOME**/lib64/R/lib''  |base directory where R looks for shared libraries by default| 
-|''**R_PREFIX**/lib64/R/lib/libRblas.so''  |symlink to chosen BLAS library (from one of the subdirectories herein)| +|''** R_HOME**/lib64/R/lib/libRblas.so''  |symlink to chosen BLAS library (from one of the subdirectories herein)| 
-|''**R_PREFIX**/lib64/R/lib/libRlapack.so''  |symlink to chosen LAPACK library (from one of the subdirectories herein)| +|''** R_HOME**/lib64/R/lib/libRlapack.so''  |symlink to chosen LAPACK library (from one of the subdirectories herein)| 
-|''**R_PREFIX**/lib64/R/lib/atlas''  |directory to hold ''libatlas.so''| +|''** R_HOME**/lib64/R/lib/atlas''  |directory to hold ''libatlas.so''| 
-|''**R_PREFIX**/lib64/R/lib/rblas''  |directory to hold the bundled ''libRblas.so'' and ''libRlapack.so'' produced by R build procedure| +|''** R_HOME**/lib64/R/lib/rblas''  |directory to hold the bundled ''libRblas.so'' and ''libRlapack.so'' produced by R build procedure| 
-|''**R_PREFIX**/lib64/R/lib/mkl''  |directory to hold MKL variants| +|''** R_HOME**/lib64/R/lib/mkl''  |directory to hold MKL variants| 
-|''**R_PREFIX**/lib64/R/lib/mkl/seq''  |directory to hold sequential MKL variant| +|''** R_HOME**/lib64/R/lib/mkl/seq''  |directory to hold sequential MKL variant| 
-|''**R_PREFIX**/lib64/R/lib/mkl/thr''  |directory to hold threaded MKL variant|+|''** R_HOME**/lib64/R/lib/mkl/thr''  |directory to hold threaded MKL variant| 
 + 
 +==== R BLAS/LAPACK ==== 
 + 
 +When we restructured the R ''lib64/R/lib'' directory, the bundled ''libRblas.so'' and ''libRlapack.so'' shared library files were moved to the ''rblas'' subdirectory.  To configure R to use its bundled libraries: 
 + 
 +<code bash> 
 +$ cd ${R_HOME}/lib64/R/lib 
 +$ rm -f libR{blas,lapack}.so 
 +$ ln -s rblas/libRblas.so . 
 +$ ln -s rblas/libRlapack.so . 
 +</code> 
 + 
 +==== ATLAS ==== 
 + 
 +The ATLAS library contains both BLAS and LAPACK APIs in a single shared library.  With ''libatlas.so'' copied into the ''** R_HOME**/lib64/R/lib/atlas'' subdirectory, we configure R to use ATLAS: 
 + 
 +<code bash> 
 +$ cd ${R_HOME}/lib64/R/lib 
 +$ rm -f libR{blas,lapack}.so 
 +$ ln -s atlas/libRblas.so . 
 +$ ln -s atlas/libRlapack.so . 
 +</code> 
 + 
 +==== Sequential MKL ==== 
 + 
 +A C source file containing a dummy function was created in ''** R_HOME**/lib64/R/lib/mkl/seq'': 
 + 
 +<file C shim.c> 
 +int 
 +mkl_shim_dummy(void) 
 +{ 
 + return 0; 
 +} 
 +</file> 
 + 
 +The shim library is then created thusly: 
 + 
 +<code bash> 
 +$ cd ${R_HOME}/lib64/R/lib/mkl/seq 
 +$ icc -shared -o libRblas.so -mkl=sequential shim.c 
 +$ ln -s libRblas.so libRlapack.so 
 +</code> 
 + 
 +To configure R to use the sequential MKL: 
 + 
 +<code bash> 
 +$ cd ${R_HOME}/lib64/R/lib 
 +$ rm -f libR{blas,lapack}.so 
 +$ ln -s mkl/seq/libRblas.so . 
 +$ ln -s mkl/seq/libRlapack.so . 
 +</code> 
 + 
 +==== Threaded MKL ==== 
 + 
 +A C source file containing a dummy function was created in ''** R_HOME**/lib64/R/lib/mkl/thr'': 
 + 
 +<file C shim.c> 
 +int 
 +mkl_shim_dummy(void) 
 +{ 
 + return 0; 
 +} 
 +</file> 
 + 
 +Since our R build used the GNU C compiler, the threaded MKL variant only works if the shim library is built against the GNU OpenMP runtime.  Using just "-mkl=parallel" links against the Intel OpenMP runtime which in testing yielded numerical issues (not actual crashes).  The shim library is then created thusly: 
 + 
 +<code bash> 
 +$ cd ${R_HOME}/lib64/R/lib/mkl/thr 
 +$ icc -shared -o libRblas.so shim.c -lmkl_gnu_thread -lmkl_core -lmkl_intel_lp64 
 +$ ln -s libRblas.so libRlapack.so 
 +</code> 
 + 
 +To configure R to use the threaded MKL: 
 + 
 +<code bash> 
 +$ cd ${R_HOME}/lib64/R/lib 
 +$ rm -f libR{blas,lapack}.so 
 +$ ln -s mkl/thr/libRblas.so . 
 +$ ln -s mkl/thr/libRlapack.so . 
 +</code>
  
 ===== Runtime-configurable substitution ===== ===== Runtime-configurable substitution =====
 +
 +By stashing each BLAS/LAPACK variant in its own subdirectory, our copy of R is actually fairly close to being runtime-configurable with respect to choice of BLAS/LAPACK.  Since all R commands will setup the environment to have the runtime linker check ''** R_HOME**/lib64/R/lib'' for shared libraries, the ''libRblas.so'' and ''libRlapack.so'' symlinks in that directory will always have priority over any other path we might add to ''LD_LIBRARY_PATH'' prior to issuing the ''R'' command, for example.  However, if ''libRblas.so'' and ''libRlapack.so'' are **not** present in that directory, the runtime linker will be forced to consult other paths present in ''LD_LIBRARY_PATH''.
 +
 +On our Caviness cluster we include no BLAS/LAPACK library symlinks in the base directory which R checks for shared libraries:
 +
 +<code bash>
 +$ cd ${R_HOME}/lib64/R/lib
 +$ rm -f libR{blas,lapack}.so
 +</code>
 +
 +In our VALET package definition for R we configure four variants that differ by BLAS/LAPACK (as a feature tag)
 +
 +<code bash>
 +$ vpkg_versions r
 +   :
 +r                The R Project for Statistical Computing
 +  3.5            alias to r/3.5.1
 +* 3.5.1          R 3.5.1 with system compilers, ATLAS
 +  3.5.1:mkl-seq  R 3.5.1 with system compilers, MKL (sequential)
 +  3.5.1:mkl-thr  R 3.5.1 with system compilers, MKL (multithread)
 +  3.5.1:rblas    R 3.5.1 with system compilers, R reference BLAS/LAPACK
 +</code>
 +
 +with ATLAS as the default (recommended) choice of underlying BLAS/LAPACK.  Each variant of the 3.5.1 version uses the same installation prefix, but adds a unique BLAS/LAPACK subdirectory to ''LD_LIBRARY_PATH'' when added to the user's environment:
 +
 +<code bash>
 +$ vpkg_require r/3.5.1
 +Adding package `r/3.5.1` to your environment
 +$ echo $LD_LIBRARY_PATH
 +/opt/shared/r/3.5.1/lib64/R/lib/atlas: ...
 +$ vpkg_rollback all
 +$ vpkg_require r/3.5.1:rblas
 +Adding package `r/3.5.1:rblas` to your environment
 +$ echo $LD_LIBRARY_PATH
 +/opt/shared/r/3.5.1/lib64/R/lib/rblas:
 +</code>
 +
 +This works fine so long as you don't attempt to install any R modules that require the BLAS/LAPACK functionality.  The R module-building environment defaults to using link search paths in the base directory (''** R_HOME**/lib64/R/lib'') and fails to find ''libRblas.so'' or ''libRlapack.so''.  This is easily fixed by altering two lines in the R installation's standard make flags file at ''** R_HOME**/lib64/R/etc/Makeconf'':
 +
 +<code>
 +BLAS_LIBS = -L"$(R_HOME)/lib$(R_ARCH)/$(R_BLAS_VARIANT)" -lRblas
 +</code>
 +
 +and
 +
 +<code>
 +LAPACK_LIBS = -L"$(R_HOME)/lib$(R_ARCH)/$(R_BLAS_VARIANT)" -lRlapack
 +</code>
 +
 +The link search added here will default to the usual path if ''R_BLAS_VARIANT'' is not set in the user's environment.  But in our VALET configuration of each of the variants, we set the appropriate relative path for ''R_BLAS_VARIANT'':
 +
 +<code bash>
 +$ vpkg_require r/3.5.1:rblas
 +Adding package `r/3.5.1:rblas` to your environment
 +$ echo $R_BLAS_VARIANT 
 +rblas
 +$ vpkg_rollback all
 +$ vpkg_require r/3.5.1:mkl-seq
 +Adding package `r/3.5.1:mkl-seq` to your environment
 +$ echo $R_BLAS_VARIANT 
 +mkl/seq
 +</code>
 +
 +We now have a runtime-configurable BLAS/LAPACK for this single installation of R, and any properly-packaged R modules should build fine against it.
 +
 +===== VALET configuration =====
 +
 +Here is the VALET configuration for our installation of R 3.5.1 with runtime-configurable BLAS/LAPACK:
 +
 +<file yaml>
 +r:
 +  description:       The R Project for Statistical Computing
 +  url:               http://www.r-project.org/
 +  prefix:            /opt/shared/r
 +
 +  default-version:   "3.5.1"
 +
 +  versions:
 +    "3.5":
 +      alias-to:      3.5.1
 +    "3.5.1":
 +      description:   R 3.5.1 with system compilers, ATLAS
 +      actions:
 +        - libdir:    lib64/R/lib
 +        - incdir:    lib64/R/include
 +        - libdir:    lib64/R/lib/atlas
 +        - variable:  R_BLAS_VARIANT
 +          operator:  set
 +          value:     atlas
 +
 +    "3.5.1:rblas":
 +      description:   R 3.5.1 with system compilers, R reference BLAS/LAPACK
 +      prefix:        3.5.1
 +      actions:
 +        - libdir:    lib64/R/lib
 +        - incdir:    lib64/R/include
 +        - libdir:    lib64/R/lib/rblas
 +        - variable:  R_BLAS_VARIANT
 +          operator:  set
 +          value:     rblas
 +
 +    "3.5.1:mkl-seq":
 +      description:   R 3.5.1 with system compilers, MKL (sequential)
 +      prefix:        3.5.1
 +      actions:
 +        - libdir:    lib64/R/lib
 +        - incdir:    lib64/R/include
 +        - libdir:    lib64/R/lib/mkl/seq
 +        - variable:  R_BLAS_VARIANT
 +          operator:  set
 +          value:     mkl/seq
 +
 +    "3.5.1:mkl-thr":
 +      description:   R 3.5.1 with system compilers, MKL (multithread)
 +      prefix:        3.5.1
 +      actions:
 +        - libdir:    lib64/R/lib
 +        - incdir:    lib64/R/include
 +        - libdir:    lib64/R/lib/mkl/thr
 +        - variable:  R_BLAS_VARIANT
 +          operator:  set
 +          value:     mkl/thr
 +</file>
  • technical/whitepaper/r-runtime-blas-lapack.1544459914.txt.gz
  • Last modified: 2018-12-10 11:38
  • by frey