Scalapack linsolve benchmark

We base this benchmark in the linsolve.f90 from the online tutorial Running on Mio Workshop presented January 25, 2011 at the Colorado School of Mines. see slides.

We get the program with

if [ ! -f "linsolve.f90" ]; then
  wget http://inside.mines.edu/mio/tutorial/libs/solvers/linsolve.f90
fi

This program reads one line to start the benchmark. The input must contain 5 numbers:

• N: order of linear system
• NPROC_ROWS: number of rows in process grid
• NPROC_COLS: number of columns in process grid
• ROW_BLOCK_SIZE: blocking size for matrix rows
• COL_BLOCK_SIZE: blocking size for matrix columns

Where ROW_BLOCK_SIZE * COL_BLOCK_SIZE cannot exceed 500*500.

For this benchmark we will set N = NPROC_ROWS*ROW_BLOCK_SIZE = NPROC_ROWS*ROW_BLOCK_SIZE. For example, to construct a one line file, in.dat, from N and the maximum block sizes of 500:

let N=3000
let ROW_BLOCK_SIZE=500
let COL_BLOCK_SIZE=500
let NPROC_ROWS=$N/$ROW_BLOCK_SIZE
let NPROC_COLS=$N/$COL_BLOCK_SIZE
echo "$N $NPROC_ROWS $NPROC_ROWS $ROW_BLOCK_SIZE $COL_BLOCK_SIZE" > in.dat

MAX_VECTOR_SIZE from 1000 to 2000
MMAX_MATRIX_SIZE from 250000 to 1000000

First set the variables

• $packages set to VALET packages
• $libs set to libraries
• $f90flags set to compiler flags

Since this test is completely contained in one Fortran 90 program you can compile with one compile, link and load with one command.

vpkg_rollback all
vpkg_devrequire $packages
 
mpif90 $f90flags -o solve linsolve.f90 $LDFLAGS $libs 

Some version of the mpif90 wrapper should be add to you environment, either explicitly or implicitly by VALET. Also VALET will set the LDFLAGS for your compile statement.

You must run the solve executable on the compute nodes, and submit with Grid Engine. You will need a script, which we will copy from /opt/shared/templates/openmpi/openmpi-ibverb.qs and modified so

• $MY_EXEC: ./solve < in.dat
• NPROC: $NPROC_ROWS*$NPROC_COLS
• vpkg_require line includes the Valet packages for the benchmark.

For example, with the packages, NPROC_ROWS and NPROC_COLS variables set as above:

let NPROC=$NPROC_ROWS*$NPROC_COLS
if [ ! -f "template.qs" ]; then
  sed -e 's/\$MY_EXE/.\/solve < in.dat/g;s/^echo "\(.*\)--"/date "+\1 %D %T--"/' \
     /opt/shared/templates/openmpi/openmpi-ibverb.qs > template.qs
  echo "new copy of template in template.qs"
fi
sed "s/NPROC/$NPROC/g;s#^vpkg_require.*#vpkg_require $packages#" template.qs > solve.qs

The file solve.qs will contain a script file to run the executable ./solve that reads the data from in.dat. Also $NPROCS is set the number of processors required on handle $NPROC_ROWS x $NPROC_COLS blocks.

There is only linear system solve, and it should take just a few seconds. We will use the under 4hr standby queue for testing. The variables name and N are set as described above.

qsub -N $name$N -l standby=1 -l h_rt=04:00:00 solve.qs

name=gcc
packages=scalapack/2.0.1-openmpi-gcc
libs="-lscalapack -llapack -lblas"
f90flags=''

name=gcc_atlas
packages='scalapack atlas'
libs="-lscalapack -llapack -lcblas -lf77blas -latlas"
f90flags=''

The documentation in /opt/shared/atlas/3.6.0-gcc/doc/LibReadme.txt gives the loader flags to use to load the atlas libraries, the are: -LLIBDIR -llapack -lcblas -lf77blas -latlas. The -L directory is in the LDFLAGS, which is supplied by VALET.

Also from the same documentation:

ATLAS does not provide a full LAPACK library.

This means the order the VALET packages are added is important. If we add atlas before scalapack, then the loader will not be able to find some LAPACK routines.

But this may not be optimal:

Just linking in ATLAS's liblapack.a first will not get you the best LAPACK
performance, mainly because LAPACK's untuned ILAENV will be used instead
of ATLAS's tuned one.

With these variables set and packages changed to

packages='scalapack atlas'

we get ld errors:

      ...
/opt/shared/atlas/3.6.0-gcc/lib/Linux_BULLDOZER/libf77blas.a(xerbla.o): In function `xerbla_':
xerbla.f:(.text+0x18): undefined reference to `s_wsfe'
      ... 

Explanation: the funcion xerbla_ in the library 'f77blas' was compiled with g77, not gfortran. So it needs the support routines in the g2c library. Knowing the name of this library wech find a copy:

  find /usr -name libg2c.a

find: `/usr/lib64/audit': Permission denied
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/32/libg2c.a
/usr/lib/gcc/x86_64-redhat-linux/3.4.6/libg2c.a

To remove these errors, change:

libs="-lscalapack -llapack -lcblas -lf77blas -latlas -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6 -lg2c"

New ld errors:

   ...
   linsolve.f90:(.text+0xd00): undefined reference to `slarnv_'
   ...

Explanation: The atlas version of lapack does not have all of the LAPACK routines.

  nm -g /opt/shared/atlas/3.6.0-gcc/lib/Linux_BULLDOZER/liblapack.a | grep slarnv_

  nm -g /opt/shared/scalapack/2.0.1-openmpi-gcc/lib/liblapack.a | grep slarnv_

                 U slarnv_
0000000000000000 T slarnv_
                 U slarnv_
                 U slarnv_

No output from first nm command.

You can copy the full atlas directory in your working direction and then follow the directions in /opt/shared/atlas/3.6.0-gcc/doc/LibReadme.txt, section:

**** GETTING A FULL LAPACK LIB ****

We call this library myatlas.

name=gcc_myatlas
packages='scalapack'
libs="-lscalapack -L./lib -llapack -lcblas -lf77blas -latlas"
f90flags=''

This requires a copy of atlas in your directory, ./lib and uses the g2c lib in the /usr/lib directory. You need to build your own copy of atlas.

Assuming you do not have a lib directory in your working directory:

  cp -a /opt/shared/atlas/3.6.0-gcc/lib/Linux_BULLDOZER lib
  ar x lib/liblapack.a
  cp /opt/shared/scalapack/2.0.1-openmpi-gcc/lib/liblapack.a lib
  ar r lib/liblapack.a *.o
  rm *.o
  cp /usr/lib/gcc/x86_64-redhat-linux/3.4.6/libg2c.a lib

Now you have a ./lib directory with your own copy of the atlas libraries, and the g2c library. You no longer need to specify atlas VALET package.

name=gcc_myptatlas
packages='scalapack'
libs="-lscalapack -L./lib -llapack -lptcblas -lptf77blas -latlas"
f90flags=''

Parallel threads will dynamically uses all the cores available at compile time (24), but only if problem size indicates they will help.

name=pgi_acml
packages=scalapack/2.0.1-openmpi-pgi
libs="-lscalapack -lacml"
f90flags=''

name=intel_mkl
packages=openmpi/1.4.4-intel64
libs="-lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64"
f90flags="-mkl"

Each test is repeated three time.

Each test is repeated three time.

Each test is repeated three time.

Each test is repeated three time.

A randomly generated matrix is solved using ScaLAPACK with different block sizes. The times are the average elapsed time in seconds, as reported by MPI_WTIME, over three batch job submissions.

There is not much difference between gcc_atlas and gcc_myatlas.

The speedup for ATLAS, MKL and ACML are relative to GCC with no special BLAS libraries.

A randomly generated matrix is solved using ScaLAPACK with different block sizes. The times are the average elapsed time in seconds, as reported by MPI_WTIME, over two batch job submissions.

Speedup for ATLAS, MKL and ACML compared to the reference GCC with no optimized library. ATLAS has two variants: myatlas has the complete LAPACK merged in, and myptatlas loads the multi-threaded versions of the libraries.

Elapsed time for ATLAS, MKL and ACML.