Differences

This shows you the differences between two versions of the page.

--- technical:recipes:ls-dyna [2024-06-07 11:46] – frey
+++ technical:recipes:ls-dyna [2024-06-07 12:51] (current) – [VALET package definition] frey
@@ Line 21: / Line 21: @@
 For this example, LS-DYNA will be installed in a workgroup's storage.  The base path ''$WORKDIR'' will be referenced in this recipe, which implies that prior to installation the ''workgroup'' command was used to spawn a workgroup shell.  The recipe assumes version R15.0.2 and the variants mentioned above.
-===== Directory Organization =====
+===== Directory organization =====
-The LS-DYNA software packages must be downloaded to the cluster.  To facilitate that, a directory must be created to hold those files:
+The LS-DYNA software packages must be downloaded to the cluster.  To facilitate that, a directory must be created to hold those files.  Rather than type-out the full path every time, the variable ''LS_DYNA_PREFIX'' will be set to the base directory containing LS-DYNA software:
 <code bash>
-$ mkdir -p "${WORKDIR}/sw/ls-dyna/attic/15.0.2"
+$ LS_DYNA_PREFIX="${WORKDIR}/sw/ls-dyna"
+$ mkdir -p "${LS_DYNA_PREFIX}/attic/15.0.2"
 </code>
@@ Line 32: / Line 33: @@
 <code bash>
-$ wget --directory-prefix="${WORKDIR}/sw/ls-dyna/attic/15.0.2" \
+$ wget --directory-prefix="${LS_DYNA_PREFIX}/attic/15.0.2" \
        --user=<USERNAME> --password=<PASSWORD> \
        '<DOWNLOAD-URL>'
@@ Line 46: / Line 47: @@
 <code bash>
-$ ls -l "${WORKDIR}/sw/ls-dyna/attic/15.0.2"
+$ ls -l "${LS_DYNA_PREFIX}/attic/15.0.2"
 total 234420
 -rw-r--r-- 1 user group 140454912 Apr  7 22:15 ls-dyna_hyb_d_R15_0_2_x64_centos79_ifort190_avx2_openmpi405.tgz_extractor.sh
@@ Line 60: / Line 61: @@
 <code bash>
-$ mkdir -p "${WORKDIR}/sw/ls-dyna/15.0.2/"{sse2,avx2,avx512}/{smp,hyb,mpp}/{double,single}
+$ mkdir -p "${LS_DYNA_PREFIX}/15.0.2/"{sse2,avx2,avx512}/{smp,hyb,mpp}/{double,single}
 </code>
@@ Line 68: / Line 69: @@
 <code bash>
-$ pushd "${WORKDIR}/sw/ls-dyna/15.0.2/avx2/hyb/single"
+$ pushd "${LS_DYNA_PREFIX}/15.0.2/avx2/hyb/single"
-$ sh "${WORKDIR}/sw/ls-dyna/attic/15.0.2/ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405.tgz_extractor.sh"
+$ sh "${LS_DYNA_PREFIX}/attic/15.0.2/ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405.tgz_extractor.sh"
 </code>
@@ Line 107: / Line 108: @@
 ==== Open MPI install ====
-The procedure here will not include commentary, but is present as succinctly as possible.  The process starts with setup of the directories and unpacking of the source code:
+The procedure is presented as succinctly as possible and starts with setup of the directories and unpacking of the source code:
 <code bash>
@@ Line 145: / Line 146: @@
 </code>
-If the configuration is successful, the software is built and installed (this will take some time):
+If the configuration is successful, the software can be built and installed (this will take some time):
 <code bash>
@@ Line 166: / Line 167: @@
 </code>
+=== VALET package definition ===
+To make this version of Open MPI available to your LS-DYNA installs, a VALET package definition file should be created.  The text ''<OPENMPI-INSTALL-DIR>'' should be replaced with the base installation path used above (e.g. ''/work/group/sw/openmpi'').
+<WRAP center round alert 80%>
+If the workgroup already has an existing ''${WORKDIR}/sw/valet/openmpi.vpkg_yaml'' file, the new version should be added to that file:  do not simply overwrite the file with the content shown below!
+</WRAP>
+For the configuration made on Caviness above:
+<file yaml openmpi.vpkg_yaml>
+openmpi:
+    description:         "Open MPI: Message-Passing Interface"
+    url:                 http://www.open-mpi.org/
+    prefix:              <OPENMPI-INSTALL-DIR>
+    versions:
+        "4.0.5:intel19,ls-dyna":
+            description:     Open MPI 4.0.5 for LS-DYNA, Intel 2019
+            dependencies:
+                - intel/2019
+                - libfabric/1.17.1
+</file>
+On DARWIN, the file is slightly different.
+<file yaml openmpi.vpkg_yaml>
+openmpi:
+    description:         "Open MPI: Message-Passing Interface"
+    url:                 http://www.open-mpi.org/
+    prefix:              <OPENMPI-INSTALL-DIR>
+    versions:
+        "4.0.5:intel19,ls-dyna":
+            description:     Open MPI 4.0.5 for LS-DYNA, Intel 2019
+            dependencies:
+                - intel/2019
+</file>
+===== VALET package definition =====
+With the necessary dependencies satisfied, a VALET package definition for LS-DYNA can be created.  Assuming this is the first time LS-DYNA is being installed, the file will need to be created as it appears below; if versions/variants are added in the future, that file should be augmented with just the new version/variant information.  The ''<LS_DYNA_PREFIX>'' text must be replaced in the package definition with the value of that environment variable that was assigned at the beginning of this recipe:
+<code bash>
+$ echo $LS_DYNA_PREFIX
+/work/group/sw/ls-dyna
+</code>
+The ''<LSTC-LICENSE-SERVER-NAME>'' text must be replaced with the IP address or DNS name of the computer that services the LSTC LS-DYNA license that was purchased.
+<file yaml ls-dyna.vpkg_yaml>
+ls-dyna:
+    prefix: <LS_DYNA_PREFIX>
+    url: http://www.lstc.com/
+    description: general-purpose finite element program simulating complex real world problems
+    actions:
+        - variable: LSTC_LICENSE
+          action: set
+          value: network
+        - variable: LSTC_LICENSE_SERVER
+          action: set
+          value: <LSTC-LICENSE-SERVER-NAME>
+        - variable: LSTC_INTERNAL_CLIENT
+          action: set
+          value: off
+        - variable: LSTC_ROOT
+          action: set
+          value: ${VALET_PATH_PREFIX}
+        - bindir: ${VALET_PATH_PREFIX}
+    default-version: "15.0.2:double,hybrid,avx2"
+    versions:
+        "15.0.2:single,hybrid,avx2":
+            description: 15.0.2, single-precision, Hybrid (SMP+MPP), AVX2
+            prefix: 15.0.2/avx2/hyb/single
+            dependencies:
+                - openmpi/4.0.5:intel19,ls-dyna
+        "15.0.2:double,hybrid,avx2":
+            description: 15.0.2, double-precision, Hybrid (SMP+MPP), AVX2
+            prefix: 15.0.2/avx2/hyb/double
+            dependencies:
+                - openmpi/4.0.5:intel19,ls-dyna
+</file>
+Loading one of these packages into the shell environment makes the ''ls-dyna'' command (the symlink created during install) available:
+<code bash>
+$ vpkg_require ls-dyna/single,hybrid,avx2
+Adding dependency `intel/2019u5` to your environment
+Adding dependency `libfabric/1.17.1` to your environment
+Adding dependency `openmpi/4.0.5:intel19,ls-dyna` to your environment
+Adding package `ls-dyna/15.0.2:single,avx2,hybrid` to your environment
+$ which ls-dyna
+/work/group/sw/ls-dyna/15.0.2/avx2/hyb/single/ls-dyna
+</code>
+===== Running jobs =====
+On the Caviness/DARWIN cluster, LS-DYNA computation must be submitted as a Slurm job.  The variants installed here use Open MPI, so the job script template at ''/opt/templates/slurm/generic/mpi/openmpi/openmpi.qs'' is a starting point for LS-DYNA jobs.  As in all cases for parallel jobs:
+  * The number of Slurm //tasks// represent the MPI ranks for a hybrid or MPP variant of LS-DYNA
+  * The number of CPUs per Slurm task represent the Open MP threads for an smp or hybrid variant of LS-DYNA
+Follow the comments in the script header to determine which flags to alter and how to do so.  The script template must be altered to add the desired version/variant of LS-DYNA to the job environment:
+<code>
+#
+# [EDIT] Do any pre-processing, staging, environment setup with VALET
+#        or explicit changes to PATH, LD_LIBRARY_PATH, etc.
+#
+vpkg_require ls-dyna/15.0.2:single,hybrid,avx2
+</code>
+Toward the end of the job script template, the LS-DYNA program is run:
+<code>
+#
+# [EDIT] Execute your MPI program
+#
+${UD_MPIRUN} ls-dyna i=my_model.k
+mpi_rc=$?
+</code>