LS-DYNA: Installation and Usage on Caviness/DARWIN

LS-DYNA is an explicit dynamics solver that simulates the extreme materials deformation response of structures to periods of severe loading. It is a commercial product, so prior to using it on an IT RCI cluster a license must be purchased. A floating (network-based) license is required for use of LS-DYNA on clusters; the user must coordinate with his/her IT support staff to have a license server present on the UD campus network.

LS-DYNA is distributed for Windows and Linux operating systems. Many build variants exist for each release, varying by:

• Numerics: the entire program is built with either single- or double-precision floating point arithmetic
• Parallelism:
  • OpenMP: 1 process of 1 or more threads
  • MPP: 1 or more processes per node, 1 or more nodes
    • Intel MPI, Open MPI, or Platform MPI
  • Hybrid: 1 or more processes of 1 or more threads per node, 1 or more nodes
    • Intel MPI, Open MPI, or Platform MPI
• CPU optimizations: vector processing instructions
  • SSE2
  • AVX2
  • AVX512

Determining which version(s) and variant(s) are needed depends on the hardware present and the computational work being done. As of mid 2024, the current version is R15.0.2 and the best baseline variants to install on Caviness/DARWIN are the hybrid AVX2 variants in both single- and double-precision numerics.

For this example, LS-DYNA will be installed in a workgroup's storage. The base path $WORKDIR will be referenced in this recipe, which implies that prior to installation the workgroup command was used to spawn a workgroup shell. The recipe assumes version R15.0.2 and the variants mentioned above.

The LS-DYNA software packages must be downloaded to the cluster. To facilitate that, a directory must be created to hold those files. Rather than type-out the full path every time, the variable LS_DYNA_PREFIX will be set to the base directory containing LS-DYNA software:

$ LS_DYNA_PREFIX="${WORKDIR}/sw/ls-dyna"
$ mkdir -p "${LS_DYNA_PREFIX}/attic/15.0.2"

(The -p ensures that any of the directories in that path that do not exist are created.) The files can be downloaded on a personal computer and uploaded to the directory just created, but they can also be downloaded directly on the cluster:

$ wget --directory-prefix="${LS_DYNA_PREFIX}/attic/15.0.2" \
       --user=<USERNAME> --password=<PASSWORD> \
       '<DOWNLOAD-URL>'

The <USERNAME> and <PASSWORD> (as well as the download site URL) are provided when a license is purchased. The <DOWNLOAD-URL> is determined by browsing the download site and copying the link that would be clicked to download to a personal computer. For example, browsing and copying the link to download R15.0.2 for hybrid Open MPI with AVX2 instructions and single-precision numerics on Linux is

https://ftp.lstc.com/user/mpp-dyna/R15.0.2/linux/x86-64/ifort_190_avx2/HYB/ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405.tgz_extractor.sh

Downloading (or uploading) both packages, the attic directory now contains:

$ ls -l "${LS_DYNA_PREFIX}/attic/15.0.2"
total 234420
-rw-r--r-- 1 user group 140454912 Apr  7 22:15 ls-dyna_hyb_d_R15_0_2_x64_centos79_ifort190_avx2_openmpi405.tgz_extractor.sh
-rw-r--r-- 1 user group  99661023 Apr  7 22:15 ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405.tgz_extractor.sh

In preparation for installing these packages, a hierarchy of directories must be created to hold the variants of R15.0.2. Even though just two specific variants will be installed, the entire hierarchy is created:

• to promote a uniform organization between all versions of the software that get installed.
• to make it easy to add missing variants in the future.

The directory name matches the version, 15.0.2, with subdirectories for each option:

$ mkdir -p "${LS_DYNA_PREFIX}/15.0.2/"{sse2,avx2,avx512}/{smp,hyb,mpp}/{double,single}

The single-precision variant is installed by changing to its subdirectory (created above) and running the appropriate installation extractor:

$ pushd "${LS_DYNA_PREFIX}/15.0.2/avx2/hyb/single"
$ sh "${LS_DYNA_PREFIX}/attic/15.0.2/ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405.tgz_extractor.sh"

There will be several files in the directory, including the LS-DYNA executable:

$ ls -l
total 154359
-rwxr-xr-x 1 user group  11402168 Mar 29 18:05 ansyscl
-rwxr-xr-x 1 user group 219543680 Mar 29 18:14 ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405
-rwxr-xr-x 1 user group    437864 Mar 29 18:14 ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405.l2a

Since the executable has a rather complicated name, a symlink is created so that the command ls-dyna is mapped to it:

$ ln -s ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405 ls-dyna
$ ls -l
total 154359
-rwxr-xr-x 1 user group  11402168 Mar 29 18:05 ansyscl
lrwxrwxrwx 1 user group        59 Jun  7 10:40 ls-dyna -> ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405
-rwxr-xr-x 1 user group 219543680 Mar 29 18:14 ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405
-rwxr-xr-x 1 user group    437864 Mar 29 18:14 ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405.l2a
$ popd

This variant of LS-DYNA R15.0.2 is now installed, but there are underlying prerequisites to its usability: the Open MPI library on which it was built must also be made available. That step will be covered in a subsequent section.

For each additional variant of R15.0.2, the steps above are repeated with the appropriate modifications to the paths and file names. All hybrid and MPP variants will require the availability of their underlying MPI library.

As mentioned, the variants installed in this recipe are built atop Open MPI. As the file names indicate (and as documented in the R15.0.2 release notes), Open MPI 4.0.5 and the Intel 2019 compiler were used to build the two variants in question. Caviness/DARWIN have Intel 2019 compilers present, but the exact release of Open MPI built with that compiler may not be present. An instance of Open MPI 4.0.5 compiled with Intel 2019 should be produced for the workgroup's usage.

The procedure is presented as succinctly as possible and starts with setup of the directories and unpacking of the source code:

$ mkdir -p "${WORKDIR}/sw/openmpi/attic"
$ wget --directory-prefix="${WORKDIR}/sw/openmpi/attic" \
    https://download.open-mpi.org/release/open-mpi/v4.0/openmpi-4.0.5.tar.bz2
$ mkdir -p "${WORKDIR}/sw/openmpi/src"
$ pushd "${WORKDIR}/sw/openmpi/src"
$ tar -xf "${WORKDIR}/sw/openmpi/attic/openmpi-4.0.5.tar.bz2"
$ cd openmpi-4.0.5
$ mkdir build-intel19-ls-dyna
$ cd build-intel19-ls-dyna
$ vpkg_rollback

For Caviness, the OFI libfabric communications library is used

$ vpkg_devrequire intel/2019 libfabric/1.17.1
$ ../configure --prefix="${WORKDIR}/sw/openmpi/4.0.5-intel19-ls-dyna" \
        --with-lustre --with-io-romio-flags=--with-file-system=ufs+nfs+lustre \
        --with-libfabric="$LIBFABRIC_PREFIX" \
        CC=icc CXX=icpc FC=ifort

versus the OS-provided UCX library used on DARWIN

$ vpkg_devrequire intel/2019
$ ../configure --prefix="${WORKDIR}/sw/openmpi/4.0.5-intel19-ls-dyna" \
        --with-hwloc=/opt/shared/hwloc/2 \
        --with-pmix=/opt/shared/pmix/3 \
        --with-libevent=/usr \
        --with-lustre \
        --with-io-romio-flags="--with-file-system=ufs+nfs+lustre" \
        CC=icc CXX=icpc FC=ifort

If the configuration is successful, the software can be built and installed (this will take some time):

$ make
$ make install
$ vpkg_rollback
$ popd

There are likely local configuration details associated with Open MPI that are effected via the Open MPI configuration files. A copy of the nearest-matching version maintained by IT RCI should be made. For example, on Caviness

$ cp -f /opt/shared/openmpi/4.0.3/etc/{openmpi,pmix}-mca-params.conf "${WORKDIR}/sw/openmpi/4.0.5-intel19-ls-dyna/etc"

versus on DARWIN

$ cp -f /opt/shared/openmpi/4.0.5/etc/{openmpi,pmix}-mca-params.conf "${WORKDIR}/sw/openmpi/4.0.5-intel19-ls-dyna/etc"

To make this version of Open MPI available to your LS-DYNA installs, a VALET package definition file should be created. The text <OPENMPI-INSTALL-DIR> should be replaced with the base installation path used above (e.g. /work/group/sw/openmpi).

For the configuration made on Caviness above:

openmpi.vpkg_yaml

openmpi:
    description:         "Open MPI: Message-Passing Interface"
    url:                 http://www.open-mpi.org/
    prefix:              <OPENMPI-INSTALL-DIR>
    versions:
        "4.0.5:intel19,ls-dyna":
            description:     Open MPI 4.0.5 for LS-DYNA, Intel 2019
            dependencies:
                - intel/2019
                - libfabric/1.17.1

On DARWIN, the file is slightly different.

openmpi.vpkg_yaml

openmpi:
    description:         "Open MPI: Message-Passing Interface"
    url:                 http://www.open-mpi.org/
    prefix:              <OPENMPI-INSTALL-DIR>
    versions:
        "4.0.5:intel19,ls-dyna":
            description:     Open MPI 4.0.5 for LS-DYNA, Intel 2019
            dependencies:
                - intel/2019

With the necessary dependencies satisfied, a VALET package definition for LS-DYNA can be created. Assuming this is the first time LS-DYNA is being installed, the file will need to be created as it appears below; if versions/variants are added in the future, that file should be augmented with just the new version/variant information. The <LS_DYNA_PREFIX> text must be replaced in the package definition with the value of that environment variable that was assigned at the beginning of this recipe:

$ echo $LS_DYNA_PREFIX
/work/group/sw/ls-dyna

The <LSTC-LICENSE-SERVER-NAME> text must be replaced with the IP address or DNS name of the computer that services the LSTC LS-DYNA license that was purchased.

ls-dyna.vpkg_yaml

ls-dyna:
    prefix: <LS_DYNA_PREFIX>
    url: http://www.lstc.com/
    description: general-purpose finite element program simulating complex real world problems

    actions:
        - variable: LSTC_LICENSE
          action: set
          value: network
        - variable: LSTC_LICENSE_SERVER
          action: set
          value: <LSTC-LICENSE-SERVER-NAME>
        - variable: LSTC_INTERNAL_CLIENT
          action: set
          value: off
        - variable: LSTC_ROOT
          action: set
          value: ${VALET_PATH_PREFIX}
        - bindir: ${VALET_PATH_PREFIX}

    default-version: "15.0.2:double,hybrid,avx2"

    versions:

        "15.0.2:single,hybrid,avx2":
            description: 15.0.2, single-precision, Hybrid (SMP+MPP), AVX2
            prefix: 15.0.2/avx2/hyb/single
            dependencies:
                - openmpi/4.0.5:intel19,ls-dyna
        "15.0.2:double,hybrid,avx2":
            description: 15.0.2, double-precision, Hybrid (SMP+MPP), AVX2
            prefix: 15.0.2/avx2/hyb/double
            dependencies:
                - openmpi/4.0.5:intel19,ls-dyna

Loading one of these packages into the shell environment makes the ls-dyna command (the symlink created during install) available:

$ vpkg_require ls-dyna/single,hybrid,avx2
Adding dependency `intel/2019u5` to your environment
Adding dependency `libfabric/1.17.1` to your environment
Adding dependency `openmpi/4.0.5:intel19,ls-dyna` to your environment
Adding package `ls-dyna/15.0.2:single,avx2,hybrid` to your environment
 
$ which ls-dyna
/work/group/sw/ls-dyna/15.0.2/avx2/hyb/single/ls-dyna

On the Caviness/DARWIN cluster, LS-DYNA computation must be submitted as a Slurm job. The variants installed here use Open MPI, so the job script template at /opt/templates/slurm/generic/mpi/openmpi/openmpi.qs is a starting point for LS-DYNA jobs. As in all cases for parallel jobs:

• The number of Slurm tasks represent the MPI ranks for a hybrid or MPP variant of LS-DYNA
• The number of CPUs per Slurm task represent the Open MP threads for an smp or hybrid variant of LS-DYNA

Follow the comments in the script header to determine which flags to alter and how to do so. The script template must be altered to add the desired version/variant of LS-DYNA to the job environment:

#
# [EDIT] Do any pre-processing, staging, environment setup with VALET
#        or explicit changes to PATH, LD_LIBRARY_PATH, etc.
#
vpkg_require ls-dyna/15.0.2:single,hybrid,avx2

Toward the end of the job script template, the LS-DYNA program is run:

#
# [EDIT] Execute your MPI program
#
${UD_MPIRUN} ls-dyna i=my_model.k
mpi_rc=$?