Packages
In VALET, a package is a software component, where "component" can mean an application, a code library, a set of man pages, or a combination of all these pieces. For example, Mathematica would be an application: all the end user needs is for the appropriate directory to be on his/her PATH so that typing mathematica starts the program1). On the other hand, a program using OpenMPI for parallelism might need both the PATH and LD_LIBRARY_PATH configured in order to function.
As newer versions of software are released, the older versions are usually not removed immediately when the new are installed on a system. Users need time to adapt their workflow to changes in the software; occasionally the changes may be radical enough (e.g. Python 2 versus 3) that the workflow simply cannot be adapted to the new version. A VALET package can have multiple versions associated with it. Using the same examples as above, the system may have Mathematica 6, 7 and 8 available to users. Likewise, OpenMPI 1.3.3 and 1.4.2 may be available, in each case in standard TCP and Myricom-enabled flavors.
VALET uses a versioned package id similar to those used in "modules". The versioned package id is a string containing two pieces, the package and the version, separated by a "/" character:
mathematica/6 mathematica/7 mathematica/8 openmpi/1.3.3 openmpi/1.3.3-mx openmpi/1.4.2 openmpi/1.4.2-mx
A special form of versioned package id references whatever version of a package is indicated to be the default version:
mathematica/default openmpi/default
The "/" and version information can also be omitted to indicate the default version of a package:
mathematica openmpi
The default version is specified in each package's configuration file – more on that later.
What Packages Are Available?
A list of available packages is displayed using the vpkg_list command:
[user@host ~]$ vpkg_list Available packages: openmpi pgi matlab
For individual packages, a list of versions is displayed using vpkg_versions:
[user@host ~]$ vpkg_versions openmpi Available versions in package (* = default version): [/opt/shared/valet/2.0/etc/openmpi.vpkg_json] openmpi Open MPI: Message-Passing Interface gcc alias to openmpi/1.8.2 intel64 alias to openmpi/1.8.2-intel64 1.8.2-intel64 Version 1.8.2, with Intel64(2015) compilers 1.8.2-gcc-4.8.3 Version 1.8.2, with GCC(4.8.3) compilers 1.8 alias to openmpi/1.8.2 * 1.8.2 Version 1.8.2, with GCC(system) compilers
The "*" indicates which version is currently the default.
A verbose description of a package or package version can be displayed using the vpkg_info command with a non-versioned package id (e.g. mathematica) or a versioned package id (e.g. mathematica/6):
[user@host ~]$ vpkg_info openmpi/default
[openmpi/1.8.2] {
Version 1.8.2, with GCC(system) compilers
prefix: /opt/shared/openmpi/1.8.2
affect dev env: <inherit>
add std paths: <inherit>
}
Applying Environment Changes from Package(s)
One or more packages are setup in the environment using the vpkg_require command:
[user@host ~]$ vpkg_require openmpi/default
Any problems encountered will result in no changes being made to the environment. For example, trying to configure two versions of OpenMPI simultaneously yields:
[user@host ~]$ vpkg_require openmpi/default openmpi/pgi-1.3.3 ERROR: unable to add versioned package: openmpi/pgi-1.3.3 conflicts with version: openmpi/pgi10-1.4.2-mx
Problems may occur due to dependencies or incompatibilities that a version of a package has w.r.t. another package:
[user@host ~]$ vpkg_require openmpi/pgi10-1.4.2 pgi/8 ERROR: unable to add versioned package: pgi/8 conflicts with version: pgi/10 [user@host ~]$ vpkg_require dummy gaussian ERROR: incompatibility detected between a previously added package and gaussian/g03.c02 [user@host ~]$ vpkg_require gaussian dummy ERROR: incompatibility detected between dummy/standard and gaussian/g03.c02
Undoing Environment Changes
VALET attempts to make a snapshot of your environment prior to each invocation of vpkg_require. To remove the changes made by your last use of vpkg_require, use the vpkg_rollback command:
[user@host ~]$ vpkg_rollback WARNING: no snapshots defined
The above message is displayed because vpkg_require has not yet been called! If you have called vpkg_require more than once, you can also rollback through more than just the last call to vpkg_require by supplying the number of rollbacks to attempt. For example,
[user@host ~]$ vpkg_rollback 2
would perform two rollbacks. To remove all changes made to your environment since the first use of vpkg_require, use the all argument:
[user@host ~]$ vpkg_rollback all
What If I Forget These Commands?
No problem! Just type vpkg_help on the cluster for a summary of the commands.
[user@host ~]$ vpkg_help
Package ids:
A package is identified by an identifier and an optional version identifer separated
by a forward slash:
matlab Default version of matlab
matlab/default Default version of matlab
matlab/R2007a R2007a version of matlab
Commands are:
vpkg_cli Invoke the UDSEC command-line utility
vpkg_list List the packages defined for UDSEC management
vpkg_versions <pkgid> List the versions available for a given package
vpkg_info <pkgid> Show information for a package (or a specific
version of a package)
vpkg_require <pkgid> {<pkgid>} Add configuration for the given packages
to the environment
vpkg_devrequire <pkgid> {<pkgid>} Add configuration for the given packages
to the environment; includes setup of the LDFLAGS
and CPPFLAGS variables with library/header paths
for the packages
vpkg_rollback {#|all} Attempt to restore the environment state that
existed before the last vpkg_require. A numerical
argument does multiple rollback operations, e.g.
"vpkg_rollback 2" will revert the last two calls
of "vpkg_require". The argument "all" reverts
every call to "vpkg_require" made in the shell.
vpkg_history Display a list of all versioned packages that
have been added to the environment.
Can I Add My Own Packages?
Sure! Version 1 of VALET exclusively used XML for specification of package files, and is documented in the next section. For VALET 2, both XML and JSON formats are available.
To augment the system-wide package files, create a .valet directory in your home directory and add your own package definition files inside it. VALET automatically looks in ~/.valet for package files. VALET will also check any directories present in the VALET_PATH environment variable2). The VALET_PATH behaves like the PATH envrionment variable:
export VALET_PATH="${HOME}/valet/etc:${WORKDIR}/sw/valet"
The paths in VALET_PATH will be consulted in the order they are specified, and VALET's default configuration directory is implicitly added to the end of the list of paths. This means you can "override" packages that are defined elsewhere. For example, if I add openmpi.vpkg to my ~/.valet directory, then vpkg_require openmpi will use the definitions in ~/.valet/openmpi.vpkg in preference to the ones defined by the system administrator in VALET's default configuration directory.