software:valet:02_packages

Packages

In VALET, a package is a software component, where "component" can mean an application, a code library, a set of man pages, or a combination of all these pieces. For example, Mathematica would be an application: all the end user needs is for the appropriate directory to be on his/her PATH so that typing mathematica starts the program1). On the other hand, a program using OpenMPI for parallelism might need both the PATH and LD_LIBRARY_PATH configured in order to function.

As newer versions of software are released, the older versions are usually not removed immediately when the new are installed on a system. Users need time to adapt their workflow to changes in the software; occasionally the changes may be radical enough (e.g. Python 2 versus 3) that the workflow simply cannot be adapted to the new version. A VALET package can have multiple versions associated with it. Using the same examples as above, the system may have Mathematica 6, 7 and 8 available to users. Likewise, OpenMPI 1.3.3 and 1.4.2 may be available, in each case in standard TCP and Myricom-enabled flavors.

VALET uses a versioned package id similar to those used in "modules". The versioned package id is a string containing two pieces, the package and the version, separated by a "/" character:

mathematica/6
mathematica/7
mathematica/8
openmpi/1.3.3
openmpi/1.3.3-mx
openmpi/1.4.2
openmpi/1.4.2-mx

A special form of versioned package id references whatever version of a package is indicated to be the default version:

mathematica/default
openmpi/default

The "/" and version information can also be omitted to indicate the default version of a package:

mathematica
openmpi

The default version is specified in each package's configuration file – more on that later.

A list of available packages is displayed using the vpkg_list command:

[user@host ~]$ vpkg_list
Available packages:
  openmpi
  pgi
  matlab

For individual packages, a list of versions is displayed using vpkg_versions:

[user@host ~]$ vpkg_versions openmpi
Available versions in package (* = default version):

[/opt/shared/valet/2.0/etc/openmpi.vpkg_json]
openmpi            Open MPI:  Message-Passing Interface
  gcc              alias to openmpi/1.8.2
  intel64          alias to openmpi/1.8.2-intel64
  1.8.2-intel64    Version 1.8.2, with Intel64(2015) compilers
  1.8.2-gcc-4.8.3  Version 1.8.2, with GCC(4.8.3) compilers
  1.8              alias to openmpi/1.8.2
* 1.8.2            Version 1.8.2, with GCC(system) compilers

The "*" indicates which version is currently the default.

A verbose description of a package or package version can be displayed using the vpkg_info command with a non-versioned package id (e.g. mathematica) or a versioned package id (e.g. mathematica/6):

[user@host ~]$ vpkg_info openmpi/default
[openmpi/1.8.2] {
  Version 1.8.2, with GCC(system) compilers
  prefix: /opt/shared/openmpi/1.8.2
  affect dev env: <inherit>
  add std paths: <inherit>
}

One or more packages are setup in the environment using the vpkg_require command:

[user@host ~]$ vpkg_require openmpi/default

Any problems encountered will result in no changes being made to the environment. For example, trying to configure two versions of OpenMPI simultaneously yields:

[user@host ~]$ vpkg_require openmpi/default openmpi/pgi-1.3.3
ERROR: unable to add versioned package: openmpi/pgi-1.3.3 conflicts with version: openmpi/pgi10-1.4.2-mx

Problems may occur due to dependencies or incompatibilities that a version of a package has w.r.t. another package:

[user@host ~]$ vpkg_require openmpi/pgi10-1.4.2 pgi/8
ERROR: unable to add versioned package: pgi/8 conflicts with version: pgi/10
[user@host ~]$ vpkg_require dummy gaussian
ERROR: incompatibility detected between a previously added package and gaussian/g03.c02
[user@host ~]$ vpkg_require gaussian dummy
ERROR: incompatibility detected between dummy/standard and gaussian/g03.c02

VALET attempts to make a snapshot of your environment prior to each invocation of vpkg_require. To remove the changes made by your last use of vpkg_require, use the vpkg_rollback command:

[user@host ~]$ vpkg_rollback 
WARNING: no snapshots defined

The above message is displayed because vpkg_require has not yet been called! If you have called vpkg_require more than once, you can also rollback through more than just the last call to vpkg_require by supplying the number of rollbacks to attempt. For example,

[user@host ~]$ vpkg_rollback 2

would perform two rollbacks. To remove all changes made to your environment since the first use of vpkg_require, use the all argument:

[user@host ~]$ vpkg_rollback all

No problem! Just type vpkg_help on the cluster for a summary of the commands.

[user@host ~]$ vpkg_help

  Package ids:
 
   A package is identified by an identifier and an optional version identifer separated
   by a forward slash:
 
     matlab                     Default version of matlab
     matlab/default             Default version of matlab
     matlab/R2007a              R2007a version of matlab
 
  Commands are:
 
   vpkg_cli                             Invoke the UDSEC command-line utility
   vpkg_list                            List the packages defined for UDSEC management
   vpkg_versions <pkgid>                List the versions available for a given package
   vpkg_info <pkgid>                    Show information for a package (or a specific
                                        version of a package)
   vpkg_require <pkgid> {<pkgid>}       Add configuration for the given packages
                                        to the environment
   vpkg_devrequire <pkgid> {<pkgid>}    Add configuration for the given packages
                                        to the environment; includes setup of the LDFLAGS
                                        and CPPFLAGS variables with library/header paths
                                        for the packages
   vpkg_rollback {#|all}                Attempt to restore the environment state that
                                        existed before the last vpkg_require.  A numerical
                                        argument does multiple rollback operations, e.g.
                                        "vpkg_rollback 2" will revert the last two calls
                                        of "vpkg_require".  The argument "all" reverts
                                        every call to "vpkg_require" made in the shell.
   vpkg_history                         Display a list of all versioned packages that
                                        have been added to the environment.

Sure! Version 1 of VALET exclusively used XML for specification of package files, and is documented in the next section. For VALET 2, both XML and JSON formats are available.

To augment the system-wide package files, create a .valet directory in your home directory and add your own package definition files inside it. VALET automatically looks in ~/.valet for package files. VALET will also check any directories present in the VALET_PATH environment variable2). The VALET_PATH behaves like the PATH envrionment variable:

  export VALET_PATH="${HOME}/valet/etc:${WORKDIR}/sw/valet"

The paths in VALET_PATH will be consulted in the order they are specified, and VALET's default configuration directory is implicitly added to the end of the list of paths. This means you can "override" packages that are defined elsewhere. For example, if I add openmpi.vpkg to my ~/.valet directory, then vpkg_require openmpi will use the definitions in ~/.valet/openmpi.vpkg in preference to the ones defined by the system administrator in VALET's default configuration directory.


Prev: Intro | Next: XML Package Files


1)
The mathematica startup script itself alters the environment further so that its own shared libraries are found properly, etc.
2)
In VALET 1 the VALET_SYSCONFDIR variable must be used, not VALET_PATH
  • software/valet/02_packages.txt
  • Last modified: 2023-12-21 10:44
  • by frey