Compiling and linking FFTW
We will use the example on: http://www.jimmo.org/example-of-one-dimensional-dft-of-real-data-with-fftw thanks to jimmo for sharing - "paying it forward".
The files rawftt.c and input_foo are from the above example. Here we follow the instructions on the web page, modified to use VALET. The last two sections are some local instructions for using OpenMP and MPI.
input_foo is trivial and will only 16 real values the first two example will run in a few miliseconds. For a substantial testing you should run on a compute node.
cd to a directory where you have write access. Then source this file.
wget http://www.jimmo.org/wp-content/uploads/2013/01/rawfft.c cat >input_foo <<'[...]' 1.419662 1.411450 1.413796 1.419662 1.419662 1.419662 1.419662 1.419662 1.434327 1.437846 1.437846 1.437846 1.441366 1.441366 1.437846 1.435500 [...]
System compiler
The system GNU Compiler Collection (GCC) is in your base environment. You can use VALET to add fftw to your base environment with the vpkg_devrequire command. The devrequire indicates you will need variables in you environment for compilation. You will need$CPPFLAGS and $LDFLAGS. To compile use the GCC compiler names, e.g., gcc, gfortran and g++.
vpkg_devrequire fftwgcc $CPPFLAGS $LDFLAGS/ C program with FFTW functions /-lfftw3 -lm
vpkg_devrequire to get the variables CPPFLAGS and LDFLAGS set. With these variables empty will get errors on every line that uses an FFTW function.
CPPFLAGS and LDFLAGS are automatically used for by make as flags to find directories with C header files and libraries. The make command will look for a .c suffix add create your program with the base name.
You just need to add the C compiler name and the library names.
export CC=gcc export LDLIBS='-lfftw3 -lm' make program
Example system gcc
Add fftw to you base environment
[(it_css:traine)@farber jimmo]$ vpkg_devrequire fftw Adding package `fftw/3.3.4` to your environment
Compile the C program with the gcc compiler:
[(it_css:traine)@farber jimmo]$ gcc $CPPFLAGS $LDFLAGS rawfft.c -lfftw3 -lm -o rawfft
Run the example with 16 input variables – real to complex.
[(it_css:traine)@farber jimmo]$ ./rawfft -n 16 input_foo > dft
Run again the the flags for an inverse transformation complex to real. This should recover original input
[(it_css:traine)@farber jimmo]$ ./rawfft -n 16 -i dft > org [(it_css:traine)@farber jimmo]$ diff input_foo org
Using a bundled compiler
The results of the vpkg_versions fftw command will list versions of both the library and compiler used to build the library. The required compiler will be added as a dependent package. You just need to know the name of the compiler, when you compile.
vpkg_devrequire/ FFTW with a compiler dependency /- / C compiler /
$CPPFLAGS $LDFLAGS/ C program with FFTW functions /-lfftw3 -lm
Example intel icc
To compile with newest Intel:
[(it_css:traine)@farber jimmo]$ vpkg_devrequire fftw/3.3.4-intel64-2016 Adding dependency `intel/2016.2.062` to your environment Adding package `fftw/3.3.4-intel64-2016` to your environment
Compile with Intel icc C compiler:
[(it_css:traine)@farber jimmo]$ icc $CPPFLAGS $LDFLAGS rawfft.c -lfftw3 -lm -o rawfft
rawfft.c(147): warning #167: argument of type "char **" is incompatible with parameter of type "char *"
while ((opt = getopt(argc, argv, "in:")) != -1) {
^
Ignore the warning, and run a small example with 16 input variables – real to complex, and check the inversion transform:
[(it_css:traine)@farber jimmo]$ ./rawfft -n 16 input_foo > dft [(it_css:traine)@farber jimmo]$ ./rawfft -n 16 -i dft > org [(it_css:traine)@farber jimmo]$ diff input_foo org
Using GNU: gcc -Wall
rawfft.c:147: warning: implicit declaration of function ‘getopt’
Using LLVM: clang
rawfft.c:147:16: warning: implicit declaration of function 'getopt'
is invalid in C99 [-Wimplicit-function-declaration]
while ((opt = getopt(argc, argv, "in:")) != -1) {
^
Thus the warning message concerns the implicitly declared the function getopt, and has nothing to do with fftw.
To explicitly declare getopt, add the line
#include <getopt.h>
to the source code. This should remove the warning.
Using an unbundled compiler
Occasionally, you may be using a compiler, or a compiler version, which does not appear on the vpkg_versions fftw list.
That is there is no VALET version compiled with you compiler.
The base vpkg_devrequire fftw is compiled with gcc and will work with any compiler on the system.
$ vpkg_versions fftw | grep '^*' * 3.3.4 Version 3.3.4 compiled with GCC(system) compilers
Example LLVM clang
To compile with LLVM (clang) C-compiler
[(it_css:traine)@farber jimmo]$ vpkg_require llvm/3.6.2 Adding dependency `gcc/4.8.3` to your environment Adding package `llvm/3.6.2` to your environment [(it_css:traine)@farber jimmo]$ vpkg_devrequire fftw Adding package `fftw/3.3.4` to your environment
Compile with clang C compiler: (The code was modified to explicitly include the getopt.h file.)
[(it_css:traine)@farber jimmo]$ clang $CPPFLAGS $LDFLAGS rawfft.c -lfftw3 -lm -o rawfft ^ [(it_css:traine)@farber jimmo]$ ./rawfft -n 16 input_foo > dft [(it_css:traine)@farber jimmo]$ ./rawfft -n 16 -i dft > org [(it_css:traine)@farber jimmo]$ diff input_foo org
Example PGI pgcc
Currently PGI compilers are unsupported on Farber. This means that IT staff will not normally build a library versions with the PGI compilers. You can still use the compiler and call libraries built with other compilers. To compile with PGI C compiler.
[(it_css:traine)@farber jimmo]$ vpkg_require pgi/16 Adding dependency `gcc/4.9.3` to your environment WARNING: The Portland compiler suite is not officially supported on Farber. WARNING: It has been made available by popular request. Adding package `pgi/16.1` to your environment [(it_css:traine)@farber jimmo]$ vpkg_devrequire fftw Adding package `fftw/3.3.4` to your environment
Compile with pgcc C compiler: (The code was modified to explicitly include the getopt.h file.)
[(it_css:traine)@farber jimmo]$ pgcc $CPPFLAGS $LDFLAGS rawfft.c -lfftw3 -lm -o rawfft ^ [(it_css:traine)@farber jimmo]$ ./rawfft -n 16 input_foo > dft [(it_css:traine)@farber jimmo]$ ./rawfft -n 16 -i dft > org [(it_css:traine)@farber jimmo]$ diff input_foo org
Multi-threaded FFTW
Two run using the -pe threads option of qsub or qlogin, you must modify your code following the directions
on Usage of Multi-threaded FFTW
For OpenMP:
- Add the include statement:
#include <omp.h> - Add the functions:
fftw_init_threads()andfftw_plan_with_nthreads(omp_get_max_threads()) - Compile as an openmp program with the added library flags:
-lfftw3_omp -lfftw3 -lm
Run it as a shared-memory OpenMP job – parallel-environment-pe-option
Currently on Farber:
[(it_css:traine)@farber jimmo]$ vpkg_devrequire fftw/3.3.4-intel64-2016 Adding dependency `intel/2016.2.062` to your environment Adding package `fftw/3.3.4-intel64-2016` to your environment [(it_css:traine)@farber jimmo]$ icc -qopenmp $CPPFLAGS $LDFLAGS -o rawfft_omp rawfft_omp.c -lfftw3_omp -lfftw3 ld: cannot find -lfftw3_omp
This means that this versions of fftw was not configured with openmp enabled. You may ask to have a VALET version built by IT staff, build a private versions for your work, or use MPI for parallelism (it works in shared-memory on the same node).
Distributed-memory FFTW with MPI
MPI parallelism is especially useful when you are transforming arrays so large that they do not fit into the memory of a single node. As with all MPI programming, this will require modifications to your code. For details see Distributed-memory FFTW with MPI
There are several VALET bundles of FFTW for different openmpi and compiler combinations. When compiling an MPI program you should use the supplied wrapper mpicc to compile. The wrappers will correctly set the MPI libraries, but you must add the FFTW flags, just as in the above examples. (You do not have to know the name of the compiler to use mpicc but you will need to know the compiler is you want to add some compiler flags)
For example, to compile with openmpi library and the INTEL compiler:
[(it_css:traine)@farber jimmo]$ vpkg_devrequire fftw/3.3.4-openmpi-1.8.2-intel64 Adding dependency `intel/2015.3.187` to your environment Adding dependency `openmpi/1.8.2-intel64` to your environment Adding package `fftw/3.3.4-openmpi-1.8.2-intel64` to your environment [(it_css:traine)@farber jimmo]$ mpicc $CPPFLAGS $LDFLAGS -o rawfft_mpi rawfft_mpi.c -lfftw3
Copy one of the openmpi templates, and modify it to add the same VALET packages you used to compile the program.
(You can use just vpkg_require instead of vpkg_devrequire in the queue script.) This way the program will have a runtime environment to match the compiled code.
Run it as a distrubuted-memory MPI job – parallel-environment-pe-option.
Using the MKL wrappers
Much of functionality included in the FFTW library is included in the intel MKL library, but with a different API. Intel provides wrapper functions to intercept the FFTW function calls and translate them to MKL function calls. This will not always work, since the libraries are not the same. There are also wrappers for the FFTW version 2 interface, for legacy programs.
You add a version of Intel and point the compiler to the include directory. See FFTW3 interface to MKL.
vpkg_require/ Version of Intel compiler /icc -mkl -I$MKLROOT/include/fftw/ C program with FFTW functions /
Example,
[(it_css:traine)@farber wrappers]$ vpkg_require intel/2016 Adding package `intel/2016.2.062` to your environment [(it_css:traine)@farber wrappers]$ icc -mkl -I$MKLROOT/include/fftw rawfft.c -o rawfft [(it_css:traine)@farber wrappers]$ ./rawfft -n 16 input_foo > dft [(it_css:traine)@farber wrappers]$ ./rawfft -n 16 -i dft > org [(it_css:traine)@farber wrappers]$ diff input_foo org