HPCC with intel compiler, MKL and base FFT

Start by downloading and extracting the hpcc-1.4.3 directory:

curl -s http://icl.cs.utk.edu/projectsfiles/hpcc/download/hpcc-1.4.3.tar.gz | tar zx

The hpcc-1.4.3 directory will have all the files you need to run the benchmark. Our job is to modify the setup for intel, and mkl on farber which uses VALET.

Copy the make file hpl/setup/Make.Linux_ATHLON_FBLAS to hpl/Make.intel-mkl

1. Comment lines beginning in MP or LA
2. Change /usr/bin/gcc to mpicc
3. Change /usr/bin/g77 to mpif77
4. Change CCFLAGS to -mkl -O3 -fno-alias -DHPCC_FFT_235
5. Change LINGFLAGS to -mkl -nofor-main

The Valet commands are

vpkg_devrequire intel
vpkg_devrequire openmpi/1.8.2-intel64

Exported variables (to set values for commented LAinc and LAlib)

export LAinc="$CPPFLAGS"
export LAlib="$LDFLAGS -nofor-main"

Make command with 4 threads

make -j 4 arch=intel-mkl

package `intel/2015.0.090` 
package `openmpi/1.8.2-intel64` 

N = 30000, NB = 200, P = 5 Q = 8

These runs need 40 processes (5 per row and 8 per column.) The same number of processes are run with 40 slots.

qacct values:

ru_wallclock 130.716      
ru_utime     5114.084     
ru_stime     39.425       
maxvmem      25.174G

package `acml/5.3.0-open64-fma4` 
package `open64/4.5` 
package `openmpi/1.4.4-open64` or  `openmpi/1.6.1-open64`

N = 72000, NB = 100, P = 12, Q = 16

nproc = 2x192   (384 slots with 192 MPI workers bound to a bulldozer core pair)

Two runs mostly differ by the use of Qlogic PSM endpoints