HPCC with open64 compiler, ACML and base FFT

Here are som modifications on Mills based on recommendations from Using ACML (AMD Core Math Library) In High Performance Computing Challenge (HPCC)

Changes to the make file hpl/setup/Make.Linux_ATHLON_FBLAS copied to hpl/Make.open64-acml

1. Comment lines beginning in MP or LA
2. Change /usr/bin/gcc to mpicc
3. Change /usr/bin/g77 to mpif77
4. Append -DHPCC_FFT_235 to CCFLAGS

The Valet commands are

vpkg_devrequire acml/5.2.0-open64-fma4
vpkg_devrequire openmpi/1.6.1-open64

Exported variables (to set values for commented LAinc and LAlib)

export LAinc="$CPPFLAGS"
export LAlib="$LDFLAGS -lacml"

Make command with 4 threads

make -j 4 arch=open64-acml

package `acml/5.2.0-open64-fma4` 
package `open64/4.5` 
package `openmpi/1.4.4-open64` 

N = 30000, NB = 100, P = 6, Q=8

These runs need 48 processes (6 per row and 8 per column.) The same number of processes are run with 48 or 96 slots.

HPCC benchmark results for two runs:

package `acml/5.3.0-open64-fma4` 
package `open64/4.5` 
package `openmpi/1.4.4-open64` or  `openmpi/1.6.1-open64` or  `openmpi/1.6.3-open64`

N = 72000, NB = 100, P = 12, Q = 16

nproc = 2x192   (384 slots with 192 MPI workers bound to a bulldozer core pair)

Runs mostly differ by the use of Qlogic PSM endpoints. For full examples of a grid engine scripts. see the files /opt/shared/templates/openmpi-ibverb.qs and /opt/shared/templates/openmpi-psm.qs.

All jobs will bind the processes by core using 2 cpus per process.

Common flags

Flags for ibverbs vs psm

The job submission command

 qsub -q standby-4h.q@@24core -l standby=1,h_rt=4:00:00 bhpcc.qs

 qconf -shgrp @24core

HPCC benchmark results for three runs:

N = 72000, NB = 100, P = 12, Q=16, NP=384

package `acml/5.2.0-open64-fma4` to your environment
package `open64/4.5` to your environment
package `openmpi/1.4.4-open64` to your environment

 package `acml/5.3.0-open64-fma4` to your environment
 package `open64/4.5` to your environment
 package `openmpi/1.6.1-open64` to your environment