==== Running your OpenFOAM application on Mills ====

Using the ''openmpi-psm.qs'' job script template found in ''/opt/shared/templates/gridengine/openmpi'' provides the best performance for a parallel MPI run for OpenFOAM applications.  Once you have copied the template for your job script (e.g. ''qsub_sowfa.qs''), then edit the following for your OpenFOAM application:

  - Modify NPROC to select the number of slots (cores); best results have been found specifying twice as many cores (a multiple of 24) and binding cores due to the shared FP unit (see steps 5 and 6).
  - Modify the email notification ''-m'' and ''-M'' according to the instructions if you want email to be sent at the end of the job, if the job is aborted, rescheduled, or suspended.
  - Setup your environment with your newly built OpenFOAM application by replacing the ''vpkg_require openmpi/1.4.4-gcc'' with your VALET package name (e.g. ''sowfa/2.0-OF2.1-gcc'')
  - Setup the MPI executable and options (arguments) for your OpenFOAM application (e.g. ''ABLSolver -parallel > log.ABLSolver.3.${JOB_ID} 2>err.ABLSolver.3'')
  - Remove ''#'' in front of ''#WANT_CPU_AFFINITY=YES''
  - Remove ''#'' in front of ''#WANT_HALF_CORES_ONLY=YES''

Now submit your job script using [[general/jobsched/exclusive|exclusive access]] to prevent other jobs from running on the same node(s) 

<code>
qsub -l exclusive=1 qsub_sowfa.qs
</code>

<note important> There is a 1.5.x Grid Engine integration error when **only** using OpenFOAM 2.1.1 compiled with ''openmpi-1.5.3'' which causes the following warning when initiating an MPI run 

<code> mca: base: component_find: unable to open /opt/shared/openfoam/2.1.
1-gcc/ThirdParty-2.1.1/platforms/linux64Gcc/openmpi-1.5.3/lib/openmpi/mca_plm_rs
hd: /opt/shared/openfoam/2.1.1-gcc/ThirdParty-2.1.1/platforms/linux64Gcc/openmpi
-1.5.3/lib/openmpi/mca_plm_rshd.so: cannot open shared object file: Permission d
enied (ignored)</code>

This is a warning and not an error. Please ignore.</note>