====== LS-DYNA:  Installation and Usage on Caviness/DARWIN ======

LS-DYNA is an explicit dynamics solver that simulates the extreme materials deformation response of structures to periods of severe loading.  It is a commercial product, so prior to using it on an IT RCI cluster a license must be purchased.  A floating (network-based) license is required for use of LS-DYNA on clusters; the user must coordinate with his/her IT support staff to have a license server present on the UD campus network.

LS-DYNA is distributed for Windows and Linux operating systems.  Many build variants exist for each release, varying by:

  * Numerics:  the entire program is built with either single- or double-precision floating point arithmetic
  * Parallelism:
    * OpenMP:  1 process of 1 or more threads
    * MPP:  1 or more processes per node, 1 or more nodes
      * Intel MPI, Open MPI, or Platform MPI
    * Hybrid:  1 or more processes of 1 or more threads per node, 1 or more nodes
      * Intel MPI, Open MPI, or Platform MPI
  * CPU optimizations: vector processing instructions
    * SSE2
    * AVX2
    * AVX512

Determining which version(s) and variant(s) are needed depends on the hardware present and the computational work being done.  As of mid 2024, the current version is R15.0.2 and the best baseline variants to install on Caviness/DARWIN are the hybrid AVX2 variants in both single- and double-precision numerics.

For this example, LS-DYNA will be installed in a workgroup's storage.  The base path ''$WORKDIR'' will be referenced in this recipe, which implies that prior to installation the ''workgroup'' command was used to spawn a workgroup shell.  The recipe assumes version R15.0.2 and the variants mentioned above.

===== Directory organization =====

The LS-DYNA software packages must be downloaded to the cluster.  To facilitate that, a directory must be created to hold those files.  Rather than type-out the full path every time, the variable ''LS_DYNA_PREFIX'' will be set to the base directory containing LS-DYNA software:

<code bash>
$ LS_DYNA_PREFIX="${WORKDIR}/sw/ls-dyna"
$ mkdir -p "${LS_DYNA_PREFIX}/attic/15.0.2"
</code>

(The ''-p'' ensures that any of the directories in that path that do not exist are created.)  The files can be downloaded on a personal computer and uploaded to the directory just created, but they can also be downloaded directly on the cluster:

<code bash>
$ wget --directory-prefix="${LS_DYNA_PREFIX}/attic/15.0.2" \
       --user=<USERNAME> --password=<PASSWORD> \
       '<DOWNLOAD-URL>'
</code>

The ''<USERNAME>'' and ''<PASSWORD>'' (as well as the download site URL) are provided when a license is purchased.  The ''<DOWNLOAD-URL>'' is determined by browsing the download site and copying the link that would be clicked to download to a personal computer.  For example, browsing and copying the link to download R15.0.2 for hybrid Open MPI with AVX2 instructions and single-precision numerics on Linux is 

<code>
https://ftp.lstc.com/user/mpp-dyna/R15.0.2/linux/x86-64/ifort_190_avx2/HYB/ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405.tgz_extractor.sh
</code>

Downloading (or uploading) both packages, the attic directory now contains:

<code bash>
$ ls -l "${LS_DYNA_PREFIX}/attic/15.0.2"
total 234420
-rw-r--r-- 1 user group 140454912 Apr  7 22:15 ls-dyna_hyb_d_R15_0_2_x64_centos79_ifort190_avx2_openmpi405.tgz_extractor.sh
-rw-r--r-- 1 user group  99661023 Apr  7 22:15 ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405.tgz_extractor.sh
</code>

In preparation for installing these packages, a hierarchy of directories must be created to hold the variants of R15.0.2.  Even though just two specific variants will be installed, the entire hierarchy is created:

  * to promote a uniform organization between all versions of the software that get installed.
  * to make it easy to add missing variants in the future.

The directory name matches the version, 15.0.2, with subdirectories for each option:

<code bash>
$ mkdir -p "${LS_DYNA_PREFIX}/15.0.2/"{sse2,avx2,avx512}/{smp,hyb,mpp}/{double,single}
</code>

==== Install a variant ====

The single-precision variant is installed by changing to its subdirectory (created above) and running the appropriate installation extractor:

<code bash>
$ pushd "${LS_DYNA_PREFIX}/15.0.2/avx2/hyb/single"
$ sh "${LS_DYNA_PREFIX}/attic/15.0.2/ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405.tgz_extractor.sh"
</code>

There will be several files in the directory, including the LS-DYNA executable:

<code bash>
$ ls -l
total 154359
-rwxr-xr-x 1 user group  11402168 Mar 29 18:05 ansyscl
-rwxr-xr-x 1 user group 219543680 Mar 29 18:14 ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405
-rwxr-xr-x 1 user group    437864 Mar 29 18:14 ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405.l2a
</code>

Since the executable has a rather complicated name, a symlink is created so that the command ''ls-dyna'' is mapped to it:

<code bash>
$ ln -s ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405 ls-dyna
$ ls -l
total 154359
-rwxr-xr-x 1 user group  11402168 Mar 29 18:05 ansyscl
lrwxrwxrwx 1 user group        59 Jun  7 10:40 ls-dyna -> ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405
-rwxr-xr-x 1 user group 219543680 Mar 29 18:14 ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405
-rwxr-xr-x 1 user group    437864 Mar 29 18:14 ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405.l2a
$ popd
</code>

This variant of LS-DYNA R15.0.2 is now installed, but there are underlying prerequisites to its usability:  the Open MPI library on which it was built must also be made available.  That step will be covered in a subsequent section.

==== Install additional variants ====

For each additional variant of R15.0.2, the steps above are repeated with the appropriate modifications to the paths and file names.  All hybrid and MPP variants will require the availability of their underlying MPI library.

===== Prerequisites =====

As mentioned, the variants installed in this recipe are built atop Open MPI.  As the file names indicate (and as documented in the R15.0.2 release notes), Open MPI 4.0.5 and the Intel 2019 compiler were used to build the two variants in question.  Caviness/DARWIN have Intel 2019 compilers present, but the exact release of Open MPI built with that compiler may not be present.  An instance of Open MPI 4.0.5 compiled with Intel 2019 should be produced for the workgroup's usage.

==== Open MPI install ====

The procedure is presented as succinctly as possible and starts with setup of the directories and unpacking of the source code:

<code bash>
$ mkdir -p "${WORKDIR}/sw/openmpi/attic"
$ wget --directory-prefix="${WORKDIR}/sw/openmpi/attic" \
    https://download.open-mpi.org/release/open-mpi/v4.0/openmpi-4.0.5.tar.bz2
$ mkdir -p "${WORKDIR}/sw/openmpi/src"
$ pushd "${WORKDIR}/sw/openmpi/src"
$ tar -xf "${WORKDIR}/sw/openmpi/attic/openmpi-4.0.5.tar.bz2"
$ cd openmpi-4.0.5
$ mkdir build-intel19-ls-dyna
$ cd build-intel19-ls-dyna
$ vpkg_rollback
</code>

For Caviness, the OFI libfabric communications library is used

<code bash>
$ vpkg_devrequire intel/2019 libfabric/1.17.1
$ ../configure --prefix="${WORKDIR}/sw/openmpi/4.0.5-intel19-ls-dyna" \
        --with-lustre --with-io-romio-flags=--with-file-system=ufs+nfs+lustre \
        --with-libfabric="$LIBFABRIC_PREFIX" \
        CC=icc CXX=icpc FC=ifort
</code>

versus the OS-provided UCX library used on DARWIN

<code bash>
$ vpkg_devrequire intel/2019
$ ../configure --prefix="${WORKDIR}/sw/openmpi/4.0.5-intel19-ls-dyna" \
        --with-hwloc=/opt/shared/hwloc/2 \
        --with-pmix=/opt/shared/pmix/3 \
        --with-libevent=/usr \
        --with-lustre \
        --with-io-romio-flags="--with-file-system=ufs+nfs+lustre" \
        CC=icc CXX=icpc FC=ifort
</code>

If the configuration is successful, the software can be built and installed (this will take some time):

<code bash>
$ make
$ make install
$ vpkg_rollback
$ popd
</code>

There are likely local configuration details associated with Open MPI that are effected via the Open MPI configuration files.  A copy of the nearest-matching version maintained by IT RCI should be made.  For example, on Caviness

<code bash>
$ cp -f /opt/shared/openmpi/4.0.3/etc/{openmpi,pmix}-mca-params.conf "${WORKDIR}/sw/openmpi/4.0.5-intel19-ls-dyna/etc"
</code>

versus on DARWIN

<code bash>
$ cp -f /opt/shared/openmpi/4.0.5/etc/{openmpi,pmix}-mca-params.conf "${WORKDIR}/sw/openmpi/4.0.5-intel19-ls-dyna/etc"
</code>

=== VALET package definition ===

To make this version of Open MPI available to your LS-DYNA installs, a VALET package definition file should be created.  The text ''<OPENMPI-INSTALL-DIR>'' should be replaced with the base installation path used above (e.g. ''/work/group/sw/openmpi'').

<WRAP center round alert 80%>
If the workgroup already has an existing ''${WORKDIR}/sw/valet/openmpi.vpkg_yaml'' file, the new version should be added to that file:  do not simply overwrite the file with the content shown below!
</WRAP>

For the configuration made on Caviness above:

<file yaml openmpi.vpkg_yaml>
openmpi:
    description:         "Open MPI: Message-Passing Interface"
    url:                 http://www.open-mpi.org/
    prefix:              <OPENMPI-INSTALL-DIR>
    versions:
        "4.0.5:intel19,ls-dyna":
            description:     Open MPI 4.0.5 for LS-DYNA, Intel 2019
            dependencies:
                - intel/2019
                - libfabric/1.17.1
</file>

On DARWIN, the file is slightly different.

<file yaml openmpi.vpkg_yaml>
openmpi:
    description:         "Open MPI: Message-Passing Interface"
    url:                 http://www.open-mpi.org/
    prefix:              <OPENMPI-INSTALL-DIR>
    versions:
        "4.0.5:intel19,ls-dyna":
            description:     Open MPI 4.0.5 for LS-DYNA, Intel 2019
            dependencies:
                - intel/2019
</file>

===== VALET package definition =====

With the necessary dependencies satisfied, a VALET package definition for LS-DYNA can be created.  Assuming this is the first time LS-DYNA is being installed, the file will need to be created as it appears below; if versions/variants are added in the future, that file should be augmented with just the new version/variant information.  The ''<LS_DYNA_PREFIX>'' text must be replaced in the package definition with the value of that environment variable that was assigned at the beginning of this recipe:

<code bash>
$ echo $LS_DYNA_PREFIX
/work/group/sw/ls-dyna
</code>

The ''<LSTC-LICENSE-SERVER-NAME>'' text must be replaced with the IP address or DNS name of the computer that services the LSTC LS-DYNA license that was purchased.

<file yaml ls-dyna.vpkg_yaml>
ls-dyna:
    prefix: <LS_DYNA_PREFIX>
    url: http://www.lstc.com/
    description: general-purpose finite element program simulating complex real world problems

    actions:
        - variable: LSTC_LICENSE
          action: set
          value: network
        - variable: LSTC_LICENSE_SERVER
          action: set
          value: <LSTC-LICENSE-SERVER-NAME>
        - variable: LSTC_INTERNAL_CLIENT
          action: set
          value: off
        - variable: LSTC_ROOT
          action: set
          value: ${VALET_PATH_PREFIX}
        - bindir: ${VALET_PATH_PREFIX}

    default-version: "15.0.2:double,hybrid,avx2"

    versions:

        "15.0.2:single,hybrid,avx2":
            description: 15.0.2, single-precision, Hybrid (SMP+MPP), AVX2
            prefix: 15.0.2/avx2/hyb/single
            dependencies:
                - openmpi/4.0.5:intel19,ls-dyna
        "15.0.2:double,hybrid,avx2":
            description: 15.0.2, double-precision, Hybrid (SMP+MPP), AVX2
            prefix: 15.0.2/avx2/hyb/double
            dependencies:
                - openmpi/4.0.5:intel19,ls-dyna
</file>

Loading one of these packages into the shell environment makes the ''ls-dyna'' command (the symlink created during install) available:

<code bash>
$ vpkg_require ls-dyna/single,hybrid,avx2
Adding dependency `intel/2019u5` to your environment
Adding dependency `libfabric/1.17.1` to your environment
Adding dependency `openmpi/4.0.5:intel19,ls-dyna` to your environment
Adding package `ls-dyna/15.0.2:single,avx2,hybrid` to your environment

$ which ls-dyna
/work/group/sw/ls-dyna/15.0.2/avx2/hyb/single/ls-dyna
</code>

===== Running jobs =====

On the Caviness/DARWIN cluster, LS-DYNA computation must be submitted as a Slurm job.  The variants installed here use Open MPI, so the job script template at ''/opt/templates/slurm/generic/mpi/openmpi/openmpi.qs'' is a starting point for LS-DYNA jobs.  As in all cases for parallel jobs:

  * The number of Slurm //tasks// represent the MPI ranks for a hybrid or MPP variant of LS-DYNA
  * The number of CPUs per Slurm task represent the Open MP threads for an smp or hybrid variant of LS-DYNA

Follow the comments in the script header to determine which flags to alter and how to do so.  The script template must be altered to add the desired version/variant of LS-DYNA to the job environment:

<code>
#
# [EDIT] Do any pre-processing, staging, environment setup with VALET
#        or explicit changes to PATH, LD_LIBRARY_PATH, etc.
#
vpkg_require ls-dyna/15.0.2:single,hybrid,avx2
</code>

Toward the end of the job script template, the LS-DYNA program is run:

<code>
#
# [EDIT] Execute your MPI program
#
${UD_MPIRUN} ls-dyna i=my_model.k
mpi_rc=$?
</code>