technical:whitepaper:r-runtime-blas-lapack

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
Last revisionBoth sides next revision
technical:whitepaper:r-runtime-blas-lapack [2018-12-10 12:02] – freytechnical:whitepaper:r-runtime-blas-lapack [2018-12-10 12:39] – [VALET configuration] frey
Line 16: Line 16:
   * [[https://stackoverflow.com/questions/29984141/does-installing-blas-atlas-mkl-openblas-will-speed-up-r-package-that-is-written]]   * [[https://stackoverflow.com/questions/29984141/does-installing-blas-atlas-mkl-openblas-will-speed-up-r-package-that-is-written]]
 The basic idea is: The basic idea is:
-  * copy ''libatlas.so'' to ''**R_PREFIX**/lib64/R/lib'' +  * copy ''libatlas.so'' to ''** R_HOME**/lib64/R/lib'' 
-  * remove ''libRblas.so'' and ''libRlapack.so'' from ''**R_PREFIX**/lib64/R/lib'' +  * remove ''libRblas.so'' and ''libRlapack.so'' from ''** R_HOME**/lib64/R/lib'' 
-  * symlink ''libRblas.so'' and ''libRlapack.so'' to ''libatlas.so'' in ''**R_PREFIX**/lib64/R/lib'' +  * symlink ''libRblas.so'' and ''libRlapack.so'' to ''libatlas.so'' in ''** R_HOME**/lib64/R/lib'' 
-This copy of R is configured to use ''**R_PREFIX**/lib64/R/lib'' to resolve shared libraries, so when executing the ''R'' command, for example, the symlinks will lead the runtime linker to the ATLAS library when resolving BLAS/LAPACK functions.+This copy of R is configured to use ''** R_HOME**/lib64/R/lib'' to resolve shared libraries, so when executing the ''R'' command, for example, the symlinks will lead the runtime linker to the ATLAS library when resolving BLAS/LAPACK functions.
  
 This scheme requires two things: This scheme requires two things:
Line 31: Line 31:
  
 ^Path  ^Description^ ^Path  ^Description^
-|''**R_PREFIX**/lib64/R/lib''  |base directory where R looks for shared libraries by default| +|''** R_HOME**/lib64/R/lib''  |base directory where R looks for shared libraries by default| 
-|''**R_PREFIX**/lib64/R/lib/libRblas.so''  |symlink to chosen BLAS library (from one of the subdirectories herein)| +|''** R_HOME**/lib64/R/lib/libRblas.so''  |symlink to chosen BLAS library (from one of the subdirectories herein)| 
-|''**R_PREFIX**/lib64/R/lib/libRlapack.so''  |symlink to chosen LAPACK library (from one of the subdirectories herein)| +|''** R_HOME**/lib64/R/lib/libRlapack.so''  |symlink to chosen LAPACK library (from one of the subdirectories herein)| 
-|''**R_PREFIX**/lib64/R/lib/atlas''  |directory to hold ''libatlas.so''| +|''** R_HOME**/lib64/R/lib/atlas''  |directory to hold ''libatlas.so''| 
-|''**R_PREFIX**/lib64/R/lib/rblas''  |directory to hold the bundled ''libRblas.so'' and ''libRlapack.so'' produced by R build procedure| +|''** R_HOME**/lib64/R/lib/rblas''  |directory to hold the bundled ''libRblas.so'' and ''libRlapack.so'' produced by R build procedure| 
-|''**R_PREFIX**/lib64/R/lib/mkl''  |directory to hold MKL variants| +|''** R_HOME**/lib64/R/lib/mkl''  |directory to hold MKL variants| 
-|''**R_PREFIX**/lib64/R/lib/mkl/seq''  |directory to hold sequential MKL variant| +|''** R_HOME**/lib64/R/lib/mkl/seq''  |directory to hold sequential MKL variant| 
-|''**R_PREFIX**/lib64/R/lib/mkl/thr''  |directory to hold threaded MKL variant|+|''** R_HOME**/lib64/R/lib/mkl/thr''  |directory to hold threaded MKL variant|
  
 ==== R BLAS/LAPACK ==== ==== R BLAS/LAPACK ====
  
-To configure R to use its bundled BLAS/LAPACK that was originally built:+When we restructured the R ''lib64/R/lib'' directory, the bundled ''libRblas.so'' and ''libRlapack.so'' shared library files were moved to the ''rblas'' subdirectory.  To configure R to use its bundled libraries:
  
 <code bash> <code bash>
-$ cd ${R_PREFIX}/lib64/R/lib+$ cd ${R_HOME}/lib64/R/lib
 $ rm -f libR{blas,lapack}.so $ rm -f libR{blas,lapack}.so
 $ ln -s rblas/libRblas.so . $ ln -s rblas/libRblas.so .
Line 53: Line 53:
 ==== ATLAS ==== ==== ATLAS ====
  
-The ATLAS library contains both BLAS and LAPACK APIs in a single shared library.  With ''libatlas.so'' copied into the ''**R_PREFIX**/lib64/R/lib/atlas'' subdirectory, we create two symlinks:+The ATLAS library contains both BLAS and LAPACK APIs in a single shared library.  With ''libatlas.so'' copied into the ''** R_HOME**/lib64/R/lib/atlas'' subdirectory, we configure R to use ATLAS:
  
 <code bash> <code bash>
-$ cd ${R_PREFIX}/lib64/R/lib/atlas +$ cd ${R_HOME}/lib64/R/lib
-$ ln -s libRblas.so libatlas.so +
-$ ln -s libRlapack.so libatlas.so +
-</code> +
- +
-To configure R to use ATLAS: +
- +
-<code bash> +
-$ cd ${R_PREFIX}/lib64/R/lib+
 $ rm -f libR{blas,lapack}.so $ rm -f libR{blas,lapack}.so
 $ ln -s atlas/libRblas.so . $ ln -s atlas/libRblas.so .
Line 70: Line 62:
 </code> </code>
  
-==== Sequential MKL shim ====+==== Sequential MKL ====
  
-A C source file containing a dummy function was created in ''**R_PREFIX**/lib64/R/lib/mkl/seq'':+A C source file containing a dummy function was created in ''** R_HOME**/lib64/R/lib/mkl/seq'':
  
 <file C shim.c> <file C shim.c>
Line 85: Line 77:
  
 <code bash> <code bash>
-$ cd ${R_PREFIX}/lib64/R/lib/mkl/seq+$ cd ${R_HOME}/lib64/R/lib/mkl/seq
 $ icc -shared -o libRblas.so -mkl=sequential shim.c $ icc -shared -o libRblas.so -mkl=sequential shim.c
 $ ln -s libRblas.so libRlapack.so $ ln -s libRblas.so libRlapack.so
Line 93: Line 85:
  
 <code bash> <code bash>
-$ cd ${R_PREFIX}/lib64/R/lib+$ cd ${R_HOME}/lib64/R/lib
 $ rm -f libR{blas,lapack}.so $ rm -f libR{blas,lapack}.so
 $ ln -s mkl/seq/libRblas.so . $ ln -s mkl/seq/libRblas.so .
Line 99: Line 91:
 </code> </code>
  
-==== Threaded MKL shim ====+==== Threaded MKL ====
  
-A C source file containing a dummy function was created in ''**R_PREFIX**/lib64/R/lib/mkl/thr'':+A C source file containing a dummy function was created in ''** R_HOME**/lib64/R/lib/mkl/thr'':
  
 <file C shim.c> <file C shim.c>
Line 114: Line 106:
  
 <code bash> <code bash>
-$ cd ${R_PREFIX}/lib64/R/lib/mkl/thr+$ cd ${R_HOME}/lib64/R/lib/mkl/thr
 $ icc -shared -o libRblas.so shim.c -lmkl_gnu_thread -lmkl_core -lmkl_intel_lp64 $ icc -shared -o libRblas.so shim.c -lmkl_gnu_thread -lmkl_core -lmkl_intel_lp64
 $ ln -s libRblas.so libRlapack.so $ ln -s libRblas.so libRlapack.so
Line 122: Line 114:
  
 <code bash> <code bash>
-$ cd ${R_PREFIX}/lib64/R/lib+$ cd ${R_HOME}/lib64/R/lib
 $ rm -f libR{blas,lapack}.so $ rm -f libR{blas,lapack}.so
 $ ln -s mkl/thr/libRblas.so . $ ln -s mkl/thr/libRblas.so .
Line 130: Line 122:
 ===== Runtime-configurable substitution ===== ===== Runtime-configurable substitution =====
  
-By stashing each BLAS/LAPACK variant in its own subdirectory our copy of R is actually fairly close to being +By stashing each BLAS/LAPACK variant in its own subdirectory, our copy of R is actually fairly close to being runtime-configurable with respect to choice of BLAS/LAPACK.  Since all R commands will setup the environment to have the runtime linker check ''** R_HOME**/lib64/R/lib'' for shared libraries, the ''libRblas.so'' and ''libRlapack.so'' symlinks in that directory will always have priority over any other path we might add to ''LD_LIBRARY_PATH'' prior to issuing the ''R'' command, for example.  However, if ''libRblas.so'' and ''libRlapack.so'' are **not** present in that directory, the runtime linker will be forced to consult other paths present in ''LD_LIBRARY_PATH''. 
 + 
 +On our Caviness cluster we include no BLAS/LAPACK library symlinks in the base directory which R checks for shared libraries: 
 + 
 +<code bash> 
 +$ cd ${R_HOME}/lib64/R/lib 
 +$ rm -f libR{blas,lapack}.so 
 +</code> 
 + 
 +In our VALET package definition for R we configure four variants that differ by BLAS/LAPACK (as a feature tag) 
 + 
 +<code bash> 
 +$ vpkg_versions r 
 +   : 
 +r                The R Project for Statistical Computing 
 +  3.5            alias to r/3.5.1 
 +* 3.5.1          R 3.5.1 with system compilers, ATLAS 
 +  3.5.1:mkl-seq  R 3.5.1 with system compilers, MKL (sequential) 
 +  3.5.1:mkl-thr  R 3.5.1 with system compilers, MKL (multithread) 
 +  3.5.1:rblas    R 3.5.1 with system compilers, R reference BLAS/LAPACK 
 +</code> 
 + 
 +with ATLAS as the default (recommended) choice of underlying BLAS/LAPACK.  Each variant of the 3.5.1 version uses the same installation prefix, but adds a unique BLAS/LAPACK subdirectory to ''LD_LIBRARY_PATH'' when added to the user's environment: 
 + 
 +<code bash> 
 +$ vpkg_require r/3.5.1 
 +Adding package `r/3.5.1` to your environment 
 +$ echo $LD_LIBRARY_PATH 
 +/opt/shared/r/3.5.1/lib64/R/lib/atlas: ... 
 +$ vpkg_rollback all 
 +$ vpkg_require r/3.5.1:rblas 
 +Adding package `r/3.5.1:rblas` to your environment 
 +$ echo $LD_LIBRARY_PATH 
 +/opt/shared/r/3.5.1/lib64/R/lib/rblas: 
 +</code> 
 + 
 +This works fine so long as you don't attempt to install any R modules that require the BLAS/LAPACK functionality.  The R module-building environment defaults to using link search paths in the base directory (''** R_HOME**/lib64/R/lib'') and fails to find ''libRblas.so'' or ''libRlapack.so''.  This is easily fixed by altering two lines in the R installation's standard make flags file at ''** R_HOME**/lib64/R/etc/Makeconf'': 
 + 
 +<code> 
 +BLAS_LIBS = -L"$(R_HOME)/lib$(R_ARCH)/$(R_BLAS_VARIANT)" -lRblas 
 +</code> 
 + 
 +and 
 + 
 +<code> 
 +LAPACK_LIBS = -L"$(R_HOME)/lib$(R_ARCH)/$(R_BLAS_VARIANT)" -lRlapack 
 +</code> 
 + 
 +The link search added here will default to the usual path if ''R_BLAS_VARIANT'' is not set in the user's environment.  But in our VALET configuration of each of the variants, we set the appropriate relative path for ''R_BLAS_VARIANT'': 
 + 
 +<code bash> 
 +$ vpkg_require r/3.5.1:rblas 
 +Adding package `r/3.5.1:rblas` to your environment 
 +$ echo $R_BLAS_VARIANT  
 +rblas 
 +$ vpkg_rollback all 
 +$ vpkg_require r/3.5.1:mkl-seq 
 +Adding package `r/3.5.1:mkl-seq` to your environment 
 +$ echo $R_BLAS_VARIANT  
 +mkl/seq 
 +</code> 
 + 
 +We now have a runtime-configurable BLAS/LAPACK for this single installation of R, and any properly-packaged R modules should build fine against it. 
 + 
 +===== VALET configuration ===== 
 + 
 +Here is the VALET configuration for our installation of R 3.5.1 with runtime-configurable BLAS/LAPACK: 
 + 
 +<file yaml> 
 +r: 
 +  description:       The R Project for Statistical Computing 
 +  url:               http://www.r-project.org/ 
 +  prefix:            /opt/shared/r 
 + 
 +  default-version:   "3.5.1" 
 + 
 +  versions: 
 +    "3.5": 
 +      alias-to:      3.5.1 
 +    "3.5.1": 
 +      description:   R 3.5.1 with system compilers, ATLAS 
 +      actions: 
 +        - libdir:    lib64/R/lib 
 +        - incdir:    lib64/R/include 
 +        - libdir:    lib64/R/lib/atlas 
 +        - variable:  R_BLAS_VARIANT 
 +          operator:  set 
 +          value:     atlas 
 + 
 +    "3.5.1:rblas": 
 +      description:   R 3.5.1 with system compilers, R reference BLAS/LAPACK 
 +      prefix:        3.5.1 
 +      actions: 
 +        - libdir:    lib64/R/lib 
 +        - incdir:    lib64/R/include 
 +        - libdir:    lib64/R/lib/rblas 
 +        - variable:  R_BLAS_VARIANT 
 +          operator:  set 
 +          value:     rblas 
 + 
 +    "3.5.1:mkl-seq": 
 +      description:   R 3.5.1 with system compilers, MKL (sequential) 
 +      prefix:        3.5.1 
 +      actions: 
 +        - libdir:    lib64/R/lib 
 +        - incdir:    lib64/R/include 
 +        - libdir:    lib64/R/lib/mkl/seq 
 +        - variable:  R_BLAS_VARIANT 
 +          operator:  set 
 +          value:     mkl/seq 
 + 
 +    "3.5.1:mkl-thr": 
 +      description:   R 3.5.1 with system compilers, MKL (multithread) 
 +      prefix:        3.5.1 
 +      actions: 
 +        - libdir:    lib64/R/lib 
 +        - incdir:    lib64/R/include 
 +        - libdir:    lib64/R/lib/mkl/thr 
 +        - variable:  R_BLAS_VARIANT 
 +          operator:  set 
 +          value:     mkl/thr 
 +</file>
  • technical/whitepaper/r-runtime-blas-lapack.txt
  • Last modified: 2018-12-10 12:40
  • by frey