technical:developer:wgss

WGSS: WorkGroup-Sponsored Software

This page documents a pilot project to increase inter-workgroup software visibility on the Caviness and DARWIN clusters. The policies and terms discussed herein are open to change for the duration of the pilot program.

IT-RCI staff maintain an extensive catalog of software under the /opt/shared file system on the clusters. This software is organized in a hierarchy by package name and version, with VALET facilitating their usage in the shell environment.

The standing policy regarding what software qualifies for management by IT-RCI is essentially one of demand: once requests for a package reach a threshold, the discussion of its support by IT-RCI can be effected. In some cases, though, a cluster workgroup may want to take upon itself the burden of managing a publically-visible software catalog. For example, several workgroups' collaborating on a software development project may want visibility without exposing their workgroup storage to all users of the cluster. This is what Workgroup-Sponsored Software (or WGSS/wgss) hopes to address.

Creating a WGSS

The workgroup seeking to provision a WGSS must submit a Research Computing High Performance Computing (HPC) Clusters Help Request and supply the following:

  1. All requests MUST come from the primary stakeholder of the workgroup or a group member designated by the primary stakeholder.
  2. The stakeholder (or designated group member acting on the stakeholder's behalf) assumes all responsibility for maintaining the WGSS.
  3. A unique, concise identifier («name») for the software project MUST be provided (e.g. "frey-dft").
  4. An anticipated maximum byte size to act as the WGSS quota MAY be provided; a default quota of 25 GiB will be effected otherwise.
  5. Please state which cluster, DARWIN and/or Caviness.

The WGSS storage will be provisioned as /opt/shared/wgss/«name» and owned by user "root" and the sponsoring workgroup. Privileges on the base directory will be set as drwxrwsr-x (02775), allowing the workgroup to create whatever directory hierarchy they deem appropriate thereunder. IT-RCI does recommend that the best-practices utilized in organizing /opt/shared be adopted in WGSS directories.

There will also be created by default a /opt/shared/wgss/«name»/valet directory for package definitions associated with the WGSS. IT-RCI will add a VALET package, wgss/«name», whose sole purpose will be the addition of the WGSS' VALET directory to the VALET search path in the shell environment. Any user seeking to utilize a specific WGSS can issue the command:

$ vpkg_require wgss/«name»

and thereafter will be able to work with packages defined in /opt/shared/wgss/«name»/valet in the current login or runtime shell. WGSS maintainers are asked to exercise good judgment in the naming and versioning of packages they provide. Choosing a «package-id»/«version-id» already provided by IT-RCI is likely to produce confusion or runtime issues with the software being provided; a feature tag can always be added to distinguish the WGSS package, e.g. «package-id»/«version-id»:«name». For example, the "frey-dft" WGSS needing a specially-built Open MPI 4.1.0 might use the VALET id openmpi/4.1.0:frey-dft to clearly distinguish it from other versions/variants.

Storage Policies

Sponsoring workgroups MUST agree to the following policies with respect to the WGSS storage:

  1. WGSS storage is granted for the sake of providing software in collaborative fashion. The storage is NOT to be used as working directories for jobs executed on the cluster, nor as a repository for data generated by jobs executed on the cluster. The storage is also NOT to be used in a high-throughput or high-turnover capacity (files altered very frequently).
  2. IT-RCI reserves the right to suspend access to the WGSS storage if usage outside the WGSS scope is suspected.
  3. IT-RCI does NOT offer support to the sponsor workgroup or other workgroups seeking to use the software present in the WGSS. It is the sole responsibility of the sponsoring workgroup to provide VALET package definitions or at the least documentation on the software's use. IT-RCI does NOT condone the modification of end users' Bash login files (.bashrc or .bash_profile) as a mechanism for adding software to the environment, so sponsor workgroups are asked to also foster this best-use policy in the software they provide.
  4. Augmentation of the WGSS storage quota may be requested via IT trouble ticket. As the sole owner of the storage hardware on which /opt/shared is resident, IT-RCI reserves the right to grant/deny such requests.
  5. WGSS storage will be subject to periodic snapshotting and replication off-site. IT-RCI reserves the right to manage those snapshots (removing if necessary) in pursuit of optimum and balanced performance of the storage hardware.
  6. The sponsor workgroup may request the removal of the WGSS at any time WITHOUT the consent of other workgroups present on the cluster.
  7. Removal of a WGSS shall encompass the deprovisioning of the granted storage and all snapshots on both production and off-site hardware.
  • technical/developer/wgss.txt
  • Last modified: 2022-04-11 11:07
  • by anita