Technical documentation

What you will find in this area are technical notes produced as UD IT builds and supports the University's various research computing systems.

Articles in this section discuss generic system administration tasks/observations.

• Summary of the June 2016 upgrade to Farber
• Summary of the issues experienced with Lustre the latter half of May, 2016
• Public documentation of the build process of Community Cluster 3
• Summary of 2019 job stall issues on Farber and mitigation
• Summary of the June 2019 expansion to Caviness
• Summary of the June 2019 update to the CUDA driver on Caviness GPU nodes
• Specifications for ongoing access to Mills file systems (2019-07-12 and on)
• Planned rebuild of Caviness first-generation OSS and NFS nodes
• Accessing Mills files via Globus GridFTP
• Alterations to behavior of the workgroup command
• Integration of TCPLinda+Gaussian on Caviness
• Per-process CPU time limits on Caviness login nodes
• Caviness 2021 Lustre expansion
• Regarding the transition to ACCESS and the XSEDE SSO HUB
• Use of the Intel oneAPI compilers on Caviness and DARWIN
• Use of UCX PML in Open MPI 4.x
• Compilation of Gaussian '16 to target NVIDIA Ampere GPUs
• UCX MR caching and large-scale MPI collectives

• Building PyQt5 in a Python virtualenv
• Building a Keras Python virtualenv
• Adding your own library of R modules in R_LIBS
• Building an Emcee and PyKlip Python virtualenv
• Building a Python virtualenv with a properly-integrated mpi4py module
• Building a TensorFlow Python virtualenv
• Building a Jupyter Notebook Python virtualenv
• Building an OpenMM Python virtualenv
• Building and using TELEMAC-MASCARET with VALET integration
• Basic software building and management
• Building VASP 6 on Caviness/DARWIN
• Managing multiple versions of revision-controlled repositories
• Using Caviness or DARWIN as a remote VisIt 3.2.x host
• Using a VNC for X11 applications
• Using TensorFlow on AMD GPUs (on DARWIN)
• Building MCFOST on Caviness
• Building a GCC 12.2 toolchain with NVIDIA/AMD offload for OpenACC
• Installing and using LS-DYNA on Caviness/DARWIN
• Installing GNNUnlock on Caviness

The following articles discuss some of the implementation details involved in tailoring the Slurm job scheduler to the Caviness HPC system.

• Revisions to Slurm Configuration v1.0.0 on Caviness
• Revisions to Slurm Configuration v1.1.2 on Caviness
• Revisions to Slurm Configuration v1.1.3 on Caviness
• Revisions to Slurm Configuration v1.1.4 on Caviness
• Revisions to Slurm Configuration v1.1.5 on Caviness
• Revisions to Slurm Configuration v2.0.0 on Caviness
• Revisions to Slurm Configuration v2.1.0 on Caviness
• Revisions to Slurm Configuration v2.2.1 on Caviness
• Revisions to Slurm Configuration v2.3.1 on Caviness
• Revisions to Slurm Configuration v2.3.2 on Caviness
• Revisions to Slurm Configuration v2.4.0 on Caviness

Additional articles documenting the configuration and use of Slurm on Caviness:

• UD job script templating for Slurm on Caviness
• UD plugin for automated creation of per-job temporary directories
• Revision to Slurm job submission to require GPU types

The following articles discuss some of the implementation details involved in tailoring the Slurm job scheduler to the DARWIN HPC system.

• Revisions to Slurm Configuration v1.0.7 on DARWIN
• Revision to Slurm Configuration v1.0.8 on DARWIN

Additional articles documenting the configuration and use of Slurm on DARWIN:

• UD job script templating for Slurm on DARWIN
• UD plugin for automated creation of per-job temporary directories
• Hermes Allocation Data Manager: Tracking SU Usage on DARWIN
• Controlling Swap Usage

The following articles discuss some of the implementation details involved in tailoring the Grid Engine job scheduler to the UD HPC systems.

• Using subordination to auto-suspend jobs
• Enhanced qlogin for X11 and group propagation
• Exclusive allocation of compute nodes
• Automated orphaned qlogin cleanup on cluster head nodes
• Adding cgroup integration to Grid Engine using prolog/epilog scripts
• Fully supporting Linux cgroups using UD's Grid Engine Cgroup Orchestrator (GECO) software

Compute nodes are provisioned and managed using the open source PERCEUS toolkit.

• Adding per-node init.d services without creating multiple VNFS images

Lustre is a high-performance parallel filesystem.

• Recovering a failed OST

The following articles discuss technical aspects of software development on UD HPC systems.

• Using file striping with the Lustre MPI-IO interfaces in Open MPI
• Learn how your HPC workgroup can organize its own software installs and make use of VALET to streamline software maintenance
• WGSS: WorkGroup-Sponsored Software on the clusters

According to Wikipedia, a white paper is an authoritative report or guide that helps readers understand an issue, solve a problem, or make a decision. Wikipedia claims that white papers are primarily used in marketing and government, but the definition applies equally well to the computing world.

Occasionally there may be performance benchmarking studies performed on the clusters or on new hardware being considered for use in the clusters. Any significant findings that can be made public will be published in the white paper area of the site.