| Both sides previous revision Previous revision Next revision | Previous revision |
| software:abaqus:farber [2018-10-09 19:37] – [Batch job] anita | software:abaqus:farber [2021-04-27 16:21] (current) – external edit 127.0.0.1 |
|---|
| ==== Serial ==== | ==== Serial ==== |
| |
| Below is a queue submission script example for Abaqus copied and modified from the serial template found in ''/opt/templates/gridengine/serial.qs''. It is important to read the comments and make the necessary changes for the example to work. Specifically, you must change ''port@lm.udel.edu'' to your Abaqus license server information. Please refer to your account sponsor on the cluster to determine if you have access to Abaqus. | Below is a queue submission script example for Abaqus copied and modified from the serial template found in ''/opt/shared/templates/gridengine/serial.qs''. It is important to read the comments and make the necessary changes for the example to work. Specifically, you must change ''port@lm.udel.edu'' to your Abaqus license server information. Please refer to your account sponsor on the cluster to determine if you have access to Abaqus. |
| |
| <code - abaqus-serial.qs> | <code - abaqus-serial.qs> |
| </code> | </code> |
| |
| <note important>A local script ''abaqus_mp_setup verbose'' will automatically setup your Abaqus runtime environment by defining options according to the queue submission script. In this example, since it is a serial job, the keyword ''cpus=1'' is defined and no parallel environment. See section [[abaqus#parallel | Parallel]] for defining a parallel runtime environment. The keyword ''interactive'' should always be used in queue submission scripts to run Abaqus, otherwise the job will be backgrounded and the queuing system will think the job has finished before it really has. </note> | <note important>A local script ''abaqus_mp_setup verbose'' will automatically setup your Abaqus runtime environment by defining options according to the queue submission script. In this example, since it is a serial job, the keyword ''cpus=1'' is defined and no parallel environment. See section [[:software:abaqus:farber#parallel | Parallel]] for defining a parallel runtime environment. The keyword ''interactive'' should always be used in queue submission scripts to run Abaqus, otherwise the job will be backgrounded and the queuing system will think the job has finished before it really has. </note> |
| |
| |
| It is necessary to decide which parallel environment you want to use for your Abaqus job. Based on the number of licenses required and the number of cores per node, it is likely that Abaqus will not parallelize well beyond 8-12 core. The OpenMP parallel environment, threads, works best using multiple cores (cpus) on a single node. Keep in mind that a license is needed for each cpu (core per node) in addition to the 5 licenses. | It is necessary to decide which parallel environment you want to use for your Abaqus job. Based on the number of licenses required and the number of cores per node, it is likely that Abaqus will not parallelize well beyond 8-12 core. The OpenMP parallel environment, threads, works best using multiple cores (cpus) on a single node. Keep in mind that a license is needed for each cpu (core per node) in addition to the 5 licenses. |
| |
| Below is a queue submission script example for Abaqus copied and modified from the OpenMP template found in ''/opt/templates/gridengine/openmp/openmp.qs'' to change ''NPROC'' to 8. It is important to read the comments and make the necessary changes for the example to work. Specifically, you must change ''port@lm.udel.edu'' to your Abaqus license server information. Please refer to your account sponsor on the cluster to determine if you have access to Abaqus. | Below is a queue submission script example for Abaqus copied and modified from the OpenMP template found in ''/opt/shared/templates/gridengine/openmp/openmp.qs'' to change ''NPROC'' to 8. It is important to read the comments and make the necessary changes for the example to work. Specifically, you must change ''port@lm.udel.edu'' to your Abaqus license server information. Please refer to your account sponsor on the cluster to determine if you have access to Abaqus. |
| |
| <code - abaqus-openmp.qs> | <code - abaqus-openmp.qs> |
| </code> | </code> |
| |
| <note important>A local script ''abaqus_mp_setup verbose'' will automatically setup your Abaqus runtime environment by defining options according to the queue submission script. In this example, since it is a parallel job based on threads, the keyword ''cpus=8'' is defined and 'mp_mode=MP' for the parallel environment. See section [[abaqus#serial | Serial]] for defining a serial runtime environment. The keyword ''interactive'' should always be used in queue submissions scripts to run Abaqus, otherwise the job will be backgrounded and the queuing system will think the job has finished before it really has. </note> | <note important>A local script ''abaqus_mp_setup verbose'' will automatically setup your Abaqus runtime environment by defining options according to the queue submission script. In this example, since it is a parallel job based on threads, the keyword ''cpus=8'' is defined and 'mp_mode=MP' for the parallel environment. See section [[:software:abaqus:farber#serial | Serial]] for defining a serial runtime environment. The keyword ''interactive'' should always be used in queue submissions scripts to run Abaqus, otherwise the job will be backgrounded and the queuing system will think the job has finished before it really has. </note> |
| |
| |
| Below is a queue submission script example for Abaqus copied and modified from the generic MPI template found in ''/opt/templates/gridengine/generic-mpi/generic-mpi.qs'' where ''NPROC'' was changed to 48. Keep in mind that this job will be assigned to as many node(s) needed for 48 cores. This may be 1 node with 48 cores or multiple nodes with less cores available. However, you are not guaranteed that all licenses will be available, so you should only be specifying the number of cores relative to the number of licenses available for your research group. Keep in mind using more than 8-12 cores on a single node has NOT shown to provide significant speedup. | Below is a queue submission script example for Abaqus copied and modified from the generic MPI template found in ''/opt/shared/templates/gridengine/generic-mpi/generic-mpi.qs'' where ''NPROC'' was changed to 48. Keep in mind that this job will be assigned to as many node(s) needed for 48 cores. This may be 1 node with 48 cores or multiple nodes with less cores available. However, you are not guaranteed that all licenses will be available, so you should only be specifying the number of cores relative to the number of licenses available for your research group. Keep in mind using more than 8-12 cores on a single node has NOT shown to provide significant speedup. |
| |
| <code - abaqus-mpi.qs> | <code - abaqus-mpi.qs> |
| </code> | </code> |
| |
| Equally as important is to specify the [[clusters/mills/runapps?s[]=mem&s[]=free#qsub-s-resource-management-options|memory resource requirements]] using ''mem_free'' and ''ram_free'' in your job script. This prevents your job from being assigned to a node unless it has enough memory available on the node before it starts and it reserves this amount of memory during the run to prevent another job from being assigned to it. If we are specifying multiple cpus in a parallel environment such threads or MPI, then we divide the amount of memory needed by the number of cpus. For example, if we need 48GB using 8 cpus, then we would specify | Equally as important is to specify the [[:abstract:farber:runjobs:schedule_jobs#resource-management-options-on-farber|memory resource requirements]] using ''mem_free'' and ''m_mem_free'' in your job script. This prevents your job from being assigned to a node unless it has enough memory available on the node before it starts and it reserves this amount of memory during the run to prevent another job from being assigned to it. If we are specifying multiple cpus in a parallel environment such threads or MPI, then we divide the amount of memory needed by the number of cpus. For example, if we need 48GB using 8 cpus, then we would specify |
| |
| <code> | <code> |
| #$ -l pe threads 8 | #$ -l pe threads 8 |
| #$ -l mem_free=6G | #$ -l mem_free=6G |
| #$ -l ram_free=6G | #$ -l m_mem_free=6G |
| </code> | </code> |
| |
| ===== Interactive job ===== | ===== Interactive job ===== |
| |
| All interactive jobs should be run on a compute node by setting your [[general/userguide/04_compute_environ?s[]=workgroup#using-workgroup-and-directories|workgroup]] and using ''qlogin''. You need to be running [[general:training#connecting|X-Windows]] and you should make sure you are in the directory where your Abaqus input and output files will be stored. These files can be very large, so you should use your ''$WORKDIR'' filesystem. Once on the compute node, you will automatically be in the same working directory, but you will need to setup your environment to run Abaqus including the license information by supplying a valid port and license server (''port@ls.udel.edu''). In the example below, we are running ABAQUS CAE on the ''traine'' account in workgroup ''it-css'' using license server ''port@ls.udel.edu'': | All interactive jobs should be run on a compute node by setting your [[:abstract:farber:app_dev:compute_env|workgroup]] and using ''qlogin''. You need to be running [[training:ssh:ssh|X-Windows]] and you should make sure you are in the directory where your Abaqus input and output files will be stored. These files can be very large, so you should use your ''$WORKDIR'' filesystem. Once on the compute node, you will automatically be in the same working directory, but you will need to setup your environment to run Abaqus including the license information by supplying a valid port and license server (''port@ls.udel.edu''). In the example below, we are running ABAQUS CAE on the ''traine'' account in workgroup ''it-css'' using license server ''port@ls.udel.edu'': |
| |
| <code bash> | <code bash> |
| [(it_css:traine)@mills ABAQUS]$ cd $WORKDIR/traine/ABAQUS | [(it_css:traine)@farber ABAQUS]$ cd $WORKDIR/traine/ABAQUS |
| [(it_css:traine)@mills ABAQUS]$ qlogin | [(it_css:traine)@farber ABAQUS]$ qlogin |
| Your job 108587 ("QLOGIN") has been submitted | Your job 108587 ("QLOGIN") has been submitted |
| waiting for interactive job to be scheduled ... | waiting for interactive job to be scheduled ... |
| Your interactive job 108587 has been successfully scheduled. | Your interactive job 108587 has been successfully scheduled. |
| Establishing /opt/shared/GridEngine/local/qlogin_ssh session to host n016 ... | Establishing /opt/shared/GridEngine/local/qlogin_ssh session to host n016 ... |
| Last login: Tue Nov 27 14:13:55 2012 from mills.mills.hpc.udel.edu | Last login: Tue Nov 27 14:13:55 2012 from farber.farber.hpc.udel.edu |
| [traine@n016 ABAQUS]$ pwd | [traine@n016 ABAQUS]$ pwd |
| /lustre/work/it_css/traine/ABAQUS | /home/work/it_css/traine/ABAQUS |
| [traine@n016 ABAQUS]$ vpkg_require abaqus | [traine@n016 ABAQUS]$ vpkg_require abaqus |
| Adding dependency `x11/RHEL6.1` to your environment | Adding dependency `x11/RHEL6.1` to your environment |
| | |
| <code bash> | <code bash> |
| [(it_css:traine)@mills ABAQUS]$ qlogin -pe threads 8 | [(it_css:traine)@farber ABAQUS]$ qlogin -pe threads 8 |
| Your job 161231 ("QLOGIN") has been submitted | Your job 161231 ("QLOGIN") has been submitted |
| waiting for interactive job to be scheduled ... | waiting for interactive job to be scheduled ... |
| Your interactive job 161231 has been successfully scheduled. | Your interactive job 161231 has been successfully scheduled. |
| Establishing /opt/shared/OpenGridScheduler/local/qlogin_ssh session to host n015 ... | Establishing /opt/shared/OpenGridScheduler/local/qlogin_ssh session to host n015 ... |
| Last login: Fri Jan 25 13:40:04 2013 from mills.mills.hpc.udel.edu | Last login: Fri Jan 25 13:40:04 2013 from farber.farber.hpc.udel.edu |
| [traine@n016 ABAQUS]$ | [traine@n016 ABAQUS]$ |
| </code> | </code> |