====== Compiling and linking FFTW ======

We will use the example on: [[http://www.jimmo.org/example-of-one-dimensional-dft-of-real-data-with-fftw]] 
thanks to jimmo for sharing - "paying it forward".

The files ''rawftt.c'' and ''input_foo'' are from the above example.  Here we follow the instructions on the web page, modified to use VALET.  The last two sections are some local instructions for using OpenMP and MPI.
<note important>The input file ''input_foo'' is trivial and will only 16 real values the first two example will run in a few miliseconds. For a substantial testing you should run on a compute node.</note>

<note tip>**Get demo to your account**:
First ''cd'' to a directory where you have write access.  Then source this file.
<code file='getdemo.sh'>
wget http://www.jimmo.org/wp-content/uploads/2013/01/rawfft.c
cat >input_foo <<'[...]'
1.419662
1.411450
1.413796
1.419662
1.419662
1.419662
1.419662
1.419662
1.434327
1.437846
1.437846
1.437846
1.441366
1.441366
1.437846
1.435500
[...]
</code>
</note>
===== System compiler =====

The system GNU Compiler Collection (GCC) is in your base environment.  You can use VALET to add **''fftw''** to your base environment with the ''vpkg_devrequire'' command.  The devrequire indicates you will need variables in you environment for compilation.  You will need''$CPPFLAGS'' and ''$LDFLAGS''.  To compile use the GCC compiler names, e.g., ''gcc'', ''gfortran'' and ''g++''.

   - ''**vpkg_devrequire fftw**''
   - ''**gcc $CPPFLAGS $LDFLAGS**'' / // C program with FFTW functions // / ''**-lfftw3 -lm**''

<note important>**error: fftw3.h: No such file or directory**
You must use ''vpkg_devrequire'' to get the variables ''CPPFLAGS'' and ''LDFLAGS'' set.  With these variables empty will get errors on every line that uses an FFTW function.
</note>

<note tip>**Using make**
The variables ''CPPFLAGS'' and ''LDFLAGS'' are automatically used for by make as flags to find directories with C header files and libraries.  The make command will look for a ''.c'' suffix add create your program with the base name.
You just need to add the C compiler name and the library names.
<code>
export CC=gcc
export LDLIBS='-lfftw3 -lm'
make program
</code>
</note>

==== Example system gcc====

Add **''fftw''** to you base environment
<code>
[(it_css:traine)@farber jimmo]$ vpkg_devrequire fftw
Adding package `fftw/3.3.4` to your environment
</code>
Compile the C program with the ''gcc'' compiler:
<code>
[(it_css:traine)@farber jimmo]$ gcc $CPPFLAGS $LDFLAGS rawfft.c -lfftw3 -lm -o rawfft
</code>
Run the example with 16 input variables -- real to complex.
<code>
[(it_css:traine)@farber jimmo]$ ./rawfft -n 16 input_foo > dft
</code>
Run again the the flags for an inverse transformation complex to real.  This should recover original input
<code>
[(it_css:traine)@farber jimmo]$ ./rawfft -n 16 -i dft > org
[(it_css:traine)@farber jimmo]$ diff input_foo org
</code>


===== Using a bundled compiler =====

The results of the **''vpkg_versions fftw''** command will list versions of both the library and compiler used to build the library. The required compiler will be added as a dependent package.  You just need to know the name of the compiler, when you compile.

   - ''**vpkg_devrequire**'' / // FFTW with a compiler dependency // / 
   - / //C compiler// / ''**$CPPFLAGS $LDFLAGS**'' / // C program with FFTW functions // / ''**-lfftw3 -lm**''

==== Example intel icc ====

To compile with newest Intel:
<code>
[(it_css:traine)@farber jimmo]$ vpkg_devrequire fftw/3.3.4-intel64-2016
Adding dependency `intel/2016.2.062` to your environment
Adding package `fftw/3.3.4-intel64-2016` to your environment
</code>
Compile with Intel ''icc'' C compiler:
<code>
[(it_css:traine)@farber jimmo]$ icc $CPPFLAGS $LDFLAGS rawfft.c -lfftw3 -lm -o rawfft
rawfft.c(147): warning #167: argument of type "char **" is incompatible with parameter of type "char *"
  	while ((opt = getopt(argc, argv, "in:")) != -1) {
                                   ^
</code>
Ignore the warning, and run a small example with 16 input variables -- real to complex, and check the inversion transform:
<code>
[(it_css:traine)@farber jimmo]$ ./rawfft -n 16 input_foo > dft
[(it_css:traine)@farber jimmo]$ ./rawfft -n 16 -i dft > org
[(it_css:traine)@farber jimmo]$ diff input_foo org
</code>
<note tip> **warning #167:**
If you encounter a warning you do not understand, try another compiler.  In this case:

Using GNU:  ''gcc -Wall''
<code>
rawfft.c:147: warning: implicit declaration of function ‘getopt’
</code>
Using LLVM:  ''clang''
<code>
rawfft.c:147:16: warning: implicit declaration of function 'getopt' 
      is invalid in C99 [-Wimplicit-function-declaration]
        while ((opt = getopt(argc, argv, "in:")) != -1) {
                      ^
</code>
Thus the warning message concerns the implicitly declared the function ''getopt'', and has nothing to do with **''fftw''**.
To explicitly declare ''getopt'', add the line 
<code>
#include <getopt.h>
</code>
to the source code.  This should remove the warning.

See [[http://man7.org/linux/man-pages/man3/getopt.3.html]]
</note>

===== Using an unbundled compiler =====

Occasionally, you may be using a compiler, or a compiler version, which does not appear on the **''vpkg_versions fftw''** list.
That is there is no VALET version compiled with you compiler.
The base **''vpkg_devrequire fftw''** is compiled with gcc and will work with any compiler on the system.  

<note important>Your code will be compiled with your compiler choice, but the FFTW library was compiled with the GCC system compiler.
You can get a one line description of the default version of FFTW with the command:
<code>
$ vpkg_versions fftw | grep '^*'
* 3.3.4                    Version 3.3.4 compiled with GCC(system) compilers
</code>
</note>

==== Example LLVM clang ====

To compile with LLVM (clang) C-compiler 
<code>
[(it_css:traine)@farber jimmo]$ vpkg_require llvm/3.6.2
Adding dependency `gcc/4.8.3` to your environment
Adding package `llvm/3.6.2` to your environment
[(it_css:traine)@farber jimmo]$ vpkg_devrequire fftw
Adding package `fftw/3.3.4` to your environment
</code>
Compile with ''clang'' C compiler: (The code was modified to explicitly include the ''getopt.h'' file.)
<code>
[(it_css:traine)@farber jimmo]$ clang $CPPFLAGS $LDFLAGS rawfft.c -lfftw3 -lm -o rawfft                                 ^
[(it_css:traine)@farber jimmo]$ ./rawfft -n 16 input_foo > dft
[(it_css:traine)@farber jimmo]$ ./rawfft -n 16 -i dft > org
[(it_css:traine)@farber jimmo]$ diff input_foo org
</code>

==== Example PGI pgcc ====

Currently PGI compilers are unsupported on Farber.  This means that IT staff will not normally build a library versions with the PGI compilers.  You can still use the compiler and call libraries built with other compilers.
To compile with PGI C compiler.
<code>
[(it_css:traine)@farber jimmo]$ vpkg_require pgi/16
Adding dependency `gcc/4.9.3` to your environment
WARNING: The Portland compiler suite is not officially supported on Farber.
WARNING: It has been made available by popular request.
Adding package `pgi/16.1` to your environment
[(it_css:traine)@farber jimmo]$ vpkg_devrequire fftw
Adding package `fftw/3.3.4` to your environment
</code>
Compile with ''pgcc'' C compiler: (The code was modified to explicitly include the ''getopt.h'' file.)
<code>
[(it_css:traine)@farber jimmo]$ pgcc $CPPFLAGS $LDFLAGS rawfft.c -lfftw3 -lm -o rawfft                                 ^
[(it_css:traine)@farber jimmo]$ ./rawfft -n 16 input_foo > dft
[(it_css:traine)@farber jimmo]$ ./rawfft -n 16 -i dft > org
[(it_css:traine)@farber jimmo]$ diff input_foo org
</code>


===== Multi-threaded FFTW =====

Two run using the ''-pe threads'' option of ''qsub'' or ''qlogin'', you must modify your code following the directions
on [[http://fftw.org/doc/Usage-of-Multi_002dthreaded-FFTW.html#Usage-of-Multi_002dthreaded-FFTW | Usage of Multi-threaded FFTW ]]

For OpenMP:

  *  Add the include statement: ''#include <omp.h>''
  *  Add the functions: ''fftw_init_threads()'' and ''fftw_plan_with_nthreads(omp_get_max_threads())''
  *  [[general/userguide/05_development#compiling-parallel-programs-that-use-openmp|Compile as an openmp program]] with the added library flags: ''-lfftw3_omp -lfftw3 -lm''

Run it as a shared-memory OpenMP job -- [[general/userguide/06_runtime_environ#parallel-environment-pe-option]]

Currently on Farber:
<code>
[(it_css:traine)@farber jimmo]$ vpkg_devrequire fftw/3.3.4-intel64-2016
Adding dependency `intel/2016.2.062` to your environment
Adding package `fftw/3.3.4-intel64-2016` to your environment

[(it_css:traine)@farber jimmo]$ icc -qopenmp $CPPFLAGS $LDFLAGS -o rawfft_omp rawfft_omp.c -lfftw3_omp -lfftw3
ld: cannot find -lfftw3_omp
</code>

This means that this versions of fftw was not configured with openmp enabled.  You may ask to have a VALET version built by IT staff, build a private versions for your work, or use MPI for parallelism (it works in shared-memory on the same node).
===== Distributed-memory FFTW with MPI =====

MPI parallelism is especially useful when you are transforming arrays so large that they do not fit into the memory of a single node.  As with all MPI programming, this will require modifications to your code. For details see
[[http://www.fftw.org/doc/Distributed_002dmemory-FFTW-with-MPI.html#Distributed_002dmemory-FFTW-with-MPI| Distributed-memory FFTW with MPI]]

There are several VALET bundles of FFTW for different openmpi and compiler combinations.  When compiling an MPI program you should use the supplied wrapper ''mpicc'' to compile.  The wrappers will correctly set the MPI libraries, but you must add the FFTW flags, just as in the above examples. (You do not have to know the name of the compiler to use ''mpicc'' but you will need to know the compiler is you want to add some compiler flags)

For example, to compile with ''openmpi'' library and the INTEL compiler:
<code>
[(it_css:traine)@farber jimmo]$ vpkg_devrequire fftw/3.3.4-openmpi-1.8.2-intel64
Adding dependency `intel/2015.3.187` to your environment
Adding dependency `openmpi/1.8.2-intel64` to your environment
Adding package `fftw/3.3.4-openmpi-1.8.2-intel64` to your environment

[(it_css:traine)@farber jimmo]$ mpicc $CPPFLAGS $LDFLAGS -o rawfft_mpi rawfft_mpi.c -lfftw3
</code>

Copy one of the ''openmpi'' templates, and modify it to add the same VALET packages you used to compile the program.
(You can use just ''vpkg_require'' instead of ''vpkg_devrequire'' in the queue script.)  This way the program will have a runtime environment to match the compiled code.
Run it as a distrubuted-memory MPI job -- [[general/userguide/06_runtime_environ#parallel-environment-pe-option]].

===== Using the MKL wrappers =====

Much of functionality included in the FFTW library is included in the intel MKL library, but with a different API.  Intel provides wrapper functions to intercept the FFTW function calls and translate them to MKL function calls.  This will not always work, since the libraries are not the same. There are also wrappers for the FFTW version 2 interface, for legacy programs.

You add a version of Intel and point the compiler to the include directory. See [[https://software.intel.com/en-us/node/522277| FFTW3 interface to MKL]].

  - ''**vpkg_require**'' / //Version of Intel compiler// /
  - ''**icc -mkl -I$MKLROOT/include/fftw**'' / //C program with FFTW functions// /

Example, 
<code>
[(it_css:traine)@farber wrappers]$ vpkg_require intel/2016
Adding package `intel/2016.2.062` to your environment
[(it_css:traine)@farber wrappers]$ icc -mkl -I$MKLROOT/include/fftw rawfft.c -o rawfft
[(it_css:traine)@farber wrappers]$ ./rawfft -n 16 input_foo > dft
[(it_css:traine)@farber wrappers]$ ./rawfft -n 16 -i dft > org
[(it_css:traine)@farber wrappers]$ diff input_foo org
</code>

===== Additional Resources =====

  * [[http://www.fftw.org/doc/|FFTW Users manual]]