====== Abaqus on Mills ======

<note important>Abaqus version 6.14-1 will not work on Lustre.  Please use the [[abstract/mills/filesystems/filesystems#archive|Archive]] filesystem on Mills  </note>

===== Batch job =====

You must define your [[abstract:mills:app_dev:compute_env|workgroup]] before submitting any job on the cluster. Due to the size of input and output files associated with Abaqus, it is suggested that separate project directories be created in ''$WORKDIR'' for each run. You will need to setup your environment for Abaqus including the license information by supplying a valid port and license server (''port@lm.udel.edu'') in your queue submission script. Examples of queue submission scripts are available for serial (''abaqus-serial.qs'') and parallel (''abaqus-openmp.qs'' and ''abaqus-mpi.qs'') batch jobs. Once you have your queue submission file setup, you simply use ''qsub abaqus-serial.qs'' to submit a serial batch job, ''qsub abaqus-openmp.qs'' to submit a parallel OpenMP batch job, or ''qsub abaqus-mpi.qs'' to submit a parallel generic MPI batch job. 

==== Serial ====

Below is a queue submission script example for Abaqus copied and modified from the serial template found in ''/opt/shared/templates/gridengine/serial.qs''.  It is important to read the comments and make the necessary changes for the example to work.  Specifically, you must change ''port@lm.udel.edu'' to your Abaqus license server information.  Please refer to your account sponsor on the cluster to determine if you have access to Abaqus.

<code - abaqus-serial.qs>
# 1. Modify the value of INPUT_FILE to be the name of your
#    Abaqus input file
# 2. Modify the value of JOBNAME to be the name of your
#    Abaqus job file = output files
#
# If you want an email message to be sent to you when your job ultimately
# finishes, edit the -M line to have your email address and change the
# next two lines to start with #$ instead of just #
# -m eas
# -M my_address@mail.server.com
#

# Name of the input file:
INPUT_FILE=sample.inp
# Name of the job and output file:
JOBNAME=sample-tst

# Setup the environment; choose the Abaqus version that's right
# for you and change port@lm.udel.edu for your Abaqus license server:
vpkg_require abaqus
export LM_LICENSE_FILE=port@lm.udel.edu

# Setup your Abaqus runtime environment based on serial, openmp or mpi; 
abaqus_mp_setup verbose

# Run the program:
if [ -n "$INPUT_FILE" ]; then
  if [ -n "$JOBNAME" ]; then
     abaqus job=$JOBNAME input=$INPUT_FILE interactive
  else
     echo "ERROR: You must set the JOBNAME variable in your queue script."
     exit 1
  fi
else
  echo "ERROR:  You must set the INPUT_FILE variable in your queue script."
  exit 1
fi
</code>

<note important>A local script ''abaqus_mp_setup verbose'' will automatically setup your Abaqus runtime environment by defining options according to the queue submission script. In this example, since it is a serial job, the keyword ''cpus=1'' is defined and no parallel environment. See section [[:software:abaqus:mills#parallel | Parallel]] for defining a parallel runtime environment. The keyword ''interactive'' should always be used in queue submission scripts to run Abaqus, otherwise the job will be backgrounded and the queuing system will think the job has finished before it really has. </note>


==== Parallel ====

The parallel portions of Abaqus are restricted to the solver and operations on the elements. Here is a list of operations with the corresponding parallel mode that Abaqus supports:

  * Element operations - MPI only 
  * Iterative solver - MPI or threads 
  * Direct solver - Threads only 
  * Lanczos solver - Threads only

It is necessary to decide which parallel environment you want to use for your Abaqus job.  Based on the number of licenses required and the number of cores per node, it is likely that Abaqus will not parallelize well beyond 8-12 core. The OpenMP parallel environment, threads, works best using multiple cores (cpus) on a single node.  Keep in mind that a license is needed for each cpu (core per node) in addition to the 5 licenses.

Below is a queue submission script example for Abaqus copied and modified from the OpenMP template found in ''/opt/shared/templates/gridengine/openmp/openmp.qs'' to change ''NPROC'' to 8.  It is important to read the comments and make the necessary changes for the example to work.  Specifically, you must change ''port@lm.udel.edu'' to your Abaqus license server information.  Please refer to your account sponsor on the cluster to determine if you have access to Abaqus.

<code - abaqus-openmp.qs>
#
# 1. Modify "NPROC" in the -pe line to reflect the number
#    of processors desired 
# 2. Modify the value of INPUT_FILE to be the name of your
#    Abaqus input file
# 3. Modify the value of JOBNAME to be the name of your
#    Abaqus job file = output files
#
#$ -pe threads 8
#
# If you want an email message to be sent to you when your job ultimately
# finishes, edit the -M line to have your email address and change the
# next two lines to start with #$ instead of just #
# -m eas
# -M my_address@mail.server.com
#

# Name of the input file:
INPUT_FILE=sample.inp
# Name of the job and output file:
JOBNAME=sample-tst

# Setup the environment; choose the Abaqus version that's right
# for you and change port@lm.udel.edu for your Abaqus license server:
vpkg_require abaqus
export LM_LICENSE_FILE=port@lm.udel.edu

# Setup your Abaqus runtime environment based on serial, openmp or mpi
abaqus_mp_setup verbose

# Run the program:
if [ -n "$INPUT_FILE" ]; then
  if [ -n "$JOBNAME" ]; then
     abaqus job=$JOBNAME input=$INPUT_FILE interactive
  else
     echo "ERROR: You must set the JOBNAME variable in your queue script."
     exit 1
  fi
else
  echo "ERROR:  You must set the INPUT_FILE variable in your queue script."
  exit 1
fi
</code>

<note important>A local script ''abaqus_mp_setup verbose'' will automatically setup your Abaqus runtime environment by defining options according to the queue submission script. In this example, since it is a parallel job based on threads, the keyword ''cpus=8'' is defined and 'mp_mode=MP' for the parallel environment. See section [[:software:abaqus:mills#serial | Serial]] for defining a serial runtime environment. The keyword ''interactive'' should always be used in queue submissions scripts to run Abaqus, otherwise the job will be backgrounded and the queuing system will think the job has finished before it really has. </note>


Below is a queue submission script example for Abaqus copied and modified from the generic MPI template found in ''/opt/shared/templates/gridengine/generic-mpi/generic-mpi.qs'' where ''NPROC'' was changed to 48. Keep in mind that this job will be assigned to as many node(s) needed for 48 cores.  This may be 1 node with 48 cores or multiple nodes with less cores available.   However, you are not guaranteed that all licenses will be available, so you should only be specifying the number of cores relative to the number of licenses available for your research group.  Keep in mind using more than 8-12 cores on a single node has NOT shown to provide significant speedup.

<code - abaqus-mpi.qs>
#
# The p.e. produces a "machines" file in the job's $TMPDIR.  $NSLOTS indicates
# the granted number of cores to be used by the job.
#
# Usage:
# 1. Modify "NPROC" in the -pe line to reflect the number of processors
#    desired
# 2. Modify the value of INPUT_FILE to be the name of your
#    Abaqus input file
# 3. Modify the value of JOBNAME to be the name of your
#    Abaqus job file = output files
#
#$ -pe generic-mpi 48
#
# If you want an email message to be sent to you when your job ultimately
# finishes, edit the -M line to have your email address and change the
# next two lines to start with #$ instead of just #
# -m eas
# -M my_address@mail.server.com
#

# Name of the input file:
INPUT_FILE=sample.inp
# Name of the job and output file:
JOBNAME=sample-tst

# Setup the environment; choose the Abaqus version that's right
# for you and change port@lm.udel.edu for your Abaqus license server:
vpkg_require abaqus
export LM_LICENSE_FILE=port@lm.udel.edu

# Setup your Abaqus runtime environment based on serial, openmp or mpi
abaqus_mp_setup verbose

# Run the program:
if [ -n "$INPUT_FILE" ]; then
  if [ -n "$JOBNAME" ]; then
     abaqus job=$JOBNAME input=$INPUT_FILE interactive
  else
     echo "ERROR: You must set the JOBNAME variable in your queue script."
     exit 1
  fi
else
  echo "ERROR:  You must set the INPUT_FILE variable in your queue script."
  exit 1
fi
</code>

==== Checking resource requirements ====

You should use ''datacheck'' to get an estimate of the resource needs for your ABAQUS run by changing your script with:

<code>
    abaqus datacheck job=$JOBNAME input=$INPUT_FILE interactive
</code>    

After the datacheck analysis is complete, the resource estimates will be shown in the ''$JOBNAME.dat'' file. An example output of the datacheck analysis is shown below:
<code>

                        A N A L Y S I S    D A T A C H E C K


                   M E M O R Y   E S T I M A T E

 PROCESS      FLOATING PT       MINIMUM MEMORY        MEMORY TO
              OPERATIONS           REQUIRED          MINIMIZE I/O
             PER ITERATION         (MBYTES)           (MBYTES)

     1          5.96E+13             2348              42049


</code>

Using the ''MEMORY TO MINIMIZE I/O (MBYTES)'' column, you can determine an adequate value for the memory parameter which should be defined in your ''abaqus_v6.env'' located in the same directory as your run. In the above example, 42049, we would round up to 43000 (~43GB), so you would define:

<code>
   memory="43000 mb" 
</code>

Equally as important is to specify the [[:abstract:mills:runjobs:runjobs#resource-management-options-on-mills|memory resource requirements]] using ''mem_free'' and ''ram_free'' in your job script.  This prevents your job from being assigned to a node unless it has enough memory available on the node before it starts and it reserves this amount of memory during the run to prevent another job from being assigned to it. If we are specifying multiple cpus in a parallel environment such threads or MPI, then we divide the amount of memory needed by the number of cpus. For example, if we need 48GB using 8 cpus, then we would specify 

<code>
   #$ -l pe threads 8
   #$ -l mem_free=6G
   #$ -l ram_free=6G
</code>

===== Interactive job =====

All interactive jobs should be run on a compute node by setting your [[:abstract:mills:app_dev:compute_env|workgroup]] and using ''qlogin''. You need to be running [[training:ssh:ssh|X-Windows]] and you should make sure you are in the directory where your Abaqus input and output files will be stored. These files can be very large, so you should use your ''$WORKDIR'' filesystem. Once on the compute node, you will automatically be in the same working directory, but you will need to setup your environment to run Abaqus including the license information by supplying a valid port and license server (''port@ls.udel.edu''). In the example below, we are running ABAQUS CAE on the ''traine'' account in workgroup ''it-css'' using license server ''port@ls.udel.edu'':

<code bash>
[(it_css:traine)@mills ABAQUS]$ cd $WORKDIR/traine/ABAQUS
[(it_css:traine)@mills ABAQUS]$ qlogin
Your job 108587 ("QLOGIN") has been submitted
waiting for interactive job to be scheduled ...
Your interactive job 108587 has been successfully scheduled.
Establishing /opt/shared/GridEngine/local/qlogin_ssh session to host n016 ...
Last login: Tue Nov 27 14:13:55 2012 from mills.mills.hpc.udel.edu
[traine@n016 ABAQUS]$ pwd
/lustre/work/it_css/traine/ABAQUS
[traine@n016 ABAQUS]$ vpkg_require abaqus
Adding dependency `x11/RHEL6.1` to your environment
Adding package `abaqus/6.11-1` to your environment
[traine@n016 ABAQUS]$ export LM_LICENSE_FILE=port@ls.udel.edu
[traine@n016 ABAQUS]$ abaqus cae -mesa &
[1] 28951
[traine@n016 ABAQUS]$ Abaqus License Manager checked out the following license(s):
"cae" release 6.11 from ls.udel.edu
<5 out of 11 licenses remain available>.
</code>

This will open the Start Session and Viewport windows to start using ABAQUS CAE. If you are prototyping a job requiring multiple cpus, then you should specify the same number of ''slots=cpus'' used for ''qlogin'' such as 8 used in the example below.
 
<code bash>
[(it_css:traine)@mills ABAQUS]$ qlogin -pe threads 8
Your job 161231 ("QLOGIN") has been submitted
waiting for interactive job to be scheduled ...
Your interactive job 161231 has been successfully scheduled.
Establishing /opt/shared/OpenGridScheduler/local/qlogin_ssh session to host n015 ...
Last login: Fri Jan 25 13:40:04 2013 from mills.mills.hpc.udel.edu
[traine@n016 ABAQUS]$
</code>