hpc documentation

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.

====== XML Package Definition Files ======

{{:software:valet:valet-icon.png?128 |}}In version 2 of VALET, packages (and their versions) can still be defined in XML files (with an augmented grammar for new features).  VALET 2 can use ''.vpkg'' files from version 1 for backward compatibility, but many version 2 features are only available using the newer XML grammar.

VALET package definitions in the XML format use the file extension ''.vpkg'' or ''.vpkg_xml''.

===== Grammar =====

A VALET package is represented in XML as a document with the following overall structure:

<code xml>
<?xml version="1.0" encoding="UTF-8"?>
<package
  xmlns="http://www.udel.edu/xml/valet/1.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.udel.edu/xml/valet/1.0
                      http://www.udel.edu/xml/valet/1.0/schema.xsd"
  id="packageid">
    :
  <version id="versionid">
    :
  </version>
</package>
</code>

A set of scalar properties can be used both in the ''package'' and ''version'' elements of the configuration.  Scalar properties are:

^Element^Value^
|''<description>text...</description>''|Human-readable description of the software package|
|''<url>hyperlink</url>''|A web site documenting the package, for example|
|''<prefix>path</prefix>''|A filesystem path containing the package or the version of the package|
|''<development-env/>''|VALET should emit ''LDFLAGS'' and ''CPPFLAGS''|
|''<no-development-env/>''|VALET should not emit ''LDFLAGS'' and ''CPPFLAGS''|
|''<standard-paths/>''|VALET should check for standard sub-paths like ''bin'' and ''lib'' under the ''prefix'' directory|
|''<no-standard-paths/>''|VALET should not check for standard sub-paths like ''bin'' and ''lib'' under the ''prefix'' directory|

Typically, the package-global ''prefix'' is set to a directory containing each version of the package, and each versioned package defines a ''prefix'' relative to that directory:
<code bash>
$ ls -l /opt/shared/openmpi
total 14
drwxr-sr-x 8 user   sysadmin 8 Aug 27 16:10 1.8.2
drwxr-sr-x 8 user   sysadmin 8 Aug 27 16:10 1.8.2-gcc-4.8.3
drwxr-sr-x 8 user   sysadmin 8 Aug 28 09:26 1.8.2-intel64
drwxrwsr-x 2 user   sysadmin 3 Aug 27 17:10 attic
</code>
The package's ''prefix'' = ''/opt/shared/openmpi'' and versions would use e.g. ''prefix'' = ''1.8.2-intel64''.  If no ''prefix'' is provided for a versioned package, then VALET implicitly uses the version id.

Versioned packages that do not specify the ''development-env'', ''no-development-env'', ''standard-paths'', or ''no-standard-paths'' properties inherit the value of their parent package.

Example:
<code xml>
<?xml version="1.0" encoding="UTF-8"?>
<package
  xmlns="http://www.udel.edu/xml/valet/1.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.udel.edu/xml/valet/1.0
                      http://www.udel.edu/xml/valet/1.0/schema.xsd"
  id="gaussian">
  <description>Gaussian Quantum Chemistry Suite</description>
  <url>http://gaussian.com/</url>
  <prefix>/opt/shared/gaussian</prefix>
  
  <no-development-env/>
  <no-standard-paths/>

  <version id="g09">
    <prefix>g09d01</prefix>
  </version>
  <version id="g09d01">
  </version>
</package>
</code>

In this example, version ''g09d01'' has a prefix path of ''/opt/shared/gaussian/g09d01'' since it did not specify a ''prefix''.  Version ''g09'' provided a relative path that is appended to the prefix of its parent package to also yield ''/opt/shared/gaussian/g09d01''.  Both versions inherit the ''development-env'' and ''standard-paths'' behavior of the parent package.

The full versioned package identifiers available in the ''gaussian'' package defined above would be:
  * ''gaussian/g09''
  * ''gaussian/g09d01''
Since the ''g09'' version targets the same executables as ''g09d01'', the ''g09'' version can be considered to be an //alias// of ''g09d01''.  VALET allows for aliased versions of packages:
<code xml>
  <version id="g09" alias-to="g09d01"/>
  <version id="g09d01">
  </version>
</code>

Once the overall versioning structure of a package has been created, the environment tests and changes associated with the package and versions of the package can be added.  The three kinds of tests/changes are:

|''dependencies''|other packages that must be present or loaded into the environment; environment variable tests that must evaluate to true for configuration to happen|
|''incompatibilities''|other packages that must NOT be present in the environment; environment variable tests that must evaluate to false for configuration to happen|
|''actions''|changes to be made to the environment if all ''dependencies'' and ''incompatibilities'' are satisfied|

If present in the package-global area of the configuration, the tests/changes affect all versions of the package and are satisfied/applied before version-specific ''dependencies''/''incompatibilities'' and ''actions''.

Example:
<code xml>
<?xml version="1.0" encoding="UTF-8"?>
<package
  xmlns="http://www.udel.edu/xml/valet/1.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.udel.edu/xml/valet/1.0
                      http://www.udel.edu/xml/valet/1.0/schema.xsd"
  id="gaussian">
  <description>Gaussian Quantum Chemistry Suite</description>
  <url>http://gaussian.com/</url>
  <prefix>/opt/shared/gaussian</prefix>
  
  <no-development-env/>
  <no-standard-paths/>
  
  <dependencies>
    <predicate stage="pre-condition">
      <variable>GAUSS_SCRDIR</variable>
      <operator>not-is-set</operator>
      <message>If GAUSS_SCRDIR is not set, the working directory will be used; you do not want that.</message>
    </predicate>
  </dependencies>
  
  <actions>
    <script shell="all" action="exec" order="success-first" succeed="0">mk-gaussian-scrdir</script>
  </actions>

  <version id="g09" alias-to="g09d01"/>
  <version id="g09d01">
    <dependencies>
      <package id="pgi/14"/>
    </dependencies>
    <actions>
      <export variable="GAUSSIAN_VERSION" action="set">G09</export>
      <script action="source" shell="sh">g09.sh</script>
    </actions>
  </version>
</package>
</code>

Given the package definition above, typing ''vpkg_require gaussian/g09'' yields the following course of events:

  - ''g09'' is an alias to ''g09d01'', so version ''g09d01'' becomes the target versioned package being configured
  - Has the ''pgi/14'' package been loaded into the environment yet?
    * If not, do all of its tests and attempt to affect all of its environment changes before proceeding
      * If that fails, this configuration fails and must exit immediately
  - The installation prefix for the software is constructed as ''/opt/shared/gaussian/g09d01''
  - The ''GAUSS_SCRDIR'' environment variable is examined
    * If it does not exist in the environment the configuration fails and the ''message'' text is printed to stdout
  - The ''mk-gaussian-scrdir'' program found in VALET's ''libexec'' directory is executed
    * If the result code returned by ''mk-gaussian-scrdir'' is zero, continue
    * Otherwise, the configuration fails and exits
  - The ''GAUSSIAN_VERSION'' environment variable is assigned the value ''G09''
  - For Bash and other sh-like shells, the script ''g09.sh'' in VALET's ''libexec'' directory is sourced
    * The return code is not checked for success or failure

If control reaches the end of this pseudo-code, then the ''gaussian/g09'' package has been successfully configured in the environment.
==== Actions ====

An XML ''actions'' element can be specified either in a package or in a version of a package and contains zero or more action elements of the following kinds:

  * //variable// -- make changes to environment variables
  * //special directories// -- shorthand for alterations to ''PATH'', ''LD_LIBRARY_PATH'', etc.
  * //script// -- source/execute a script
  * //shell alias// -- set/unset an alias

Each action is represented as an XML element.  Actions attached to the package itself will be performed when //any// version of the package is selected for addition to the environment, and in addition to actions associated with the version itself.

Actions will be performed in the order they are specified in the block.

Example:
<code xml>
  <actions>
    <incdir>/opt/include</incdir>
    <export variable="PATH" action="scrub-path">/opt/bin</export>
    <bindir>/opt/bin</bindir>
    <libdir>/opt/lib</libdir>
    <libdir>/opt/lib64</libdir>
    <mandir>/opt/share/man</mandir>
    <export variable="USER_SW_PATH">/opt/bin</export>
    <export variable="CFLAGS" action="append-space" development-env="false">-O3 -g</export>
  </actions>
</code>

=== Variable action ===

Variable actions alter the value of an environment variable.  A variable action is an XML ''export'' element with the following attributes:

^Attribute^Value^Optional^
|''variable''|Environment variable name|required|
|''action''|''set'' ''unset'' ''append'' ''prepend'' ''append-path'' ''prepend-path'' ''append-space'' ''prepend-space'' ''scrub'' ''scrub-path''|default = ''set''|
|''development-env''|Only set this variable when ''vpkg_devrequire'' is used|default = no|

The text inside the ''export'' element is the //value// to add/remove/etc.

The actions are self-explanatory for the most part.  The ''scrub'' action removes all occurrences of //value// from the variable's current value in the environment.  The ''scrub-path'' action treats the value of ''variable'' as a search path and removes from that search path all occurrences (by exact match) of //value//.

The //value// may contain references to other environment variables using the syntax ''${ENV VAR NAME}''.

Example:
<code xml>
<export variable="PATH" action="scrub-path">/opt/bin</export>
</code>

=== Special directories ===

All special-directory actions are enclosed in an XML element within the ''actions'' element.  These actions are shortcuts that produce variable actions that affect standard environment variables (like ''PATH'' or ''LD_LIBRARY_PATH'').  The following XML element tags are available:

^Element Tag^Value^
|''bindir''|Paths to add to ''PATH''|
|''libdir''|Paths to add to ''LD_LIBRARY_PATH'' (and ''LDFLAGS'' under ''vpkg_devrequire'')|
|''incdir''|Paths to add to ''CPPFLAGS'' under ''vpkg_devrequire''|
|''mandir''|Paths to add to ''MANPATH''|
|''infodir''|Paths to add to ''INFOPATH''|
|''pkgconfigdir''|Paths to add to ''PKG_CONFIG_PATH''|

The text inside the XML element is the path to be added to the variable in question.  Multiple elements of the same tag are allowed.

Example:
<code xml>
<bindir>/opt/bin</bindir>
<libdir>/opt/lib</libdir>
<libdir>/opt/lib64</libdir>
<mandir>/opt/share/man</mandir>
</code>

=== Scripts ===

At times VALET may not be able to fully express the modifications necessary for a package, or the package includes its own extensive environment configuration scripts.  In such instances, a VALET package can be configured to simply execute (or source) that external script.  A script action is an XML ''script'' element with the following attributes:

^Attribute^Value^
|''action''|''source'' or ''exec''|
|''order''|''failure-first'' or ''success-first''|
|''success''|integer return code indicating successful execution/sourcing|
|''failure''|integer return code indicating failed execution/sourcing|
|''shell''|shell type for which this script is executed|

The text inside the XML ''script'' element is the path to the script to be sourced/executed.  Multiple ''script'' elements are allowed.

The ''order'' determines which is tested first, the success return code or the failure return code.  If no ''success'' or ''failure'' are specified, then the return code is not tested.  Many shell scripts return code 0 on success and a non-zero value in case of a failure; this could be tested by supplying ''order'' = ''success_first'' and ''success'' = 0.

Example:
<code xml>
<script action="source" order="success-first" success="0" shell="sh">/opt/mathematica/bin/remote_kernel.sh</script>
<script action="source" order="success-first" success="0" shell="csh">/opt/mathematica/bin/remote_kernel.csh</script>
</code>

=== Shell Aliases ===

An //alias// is a word that represents an arbitrarily complex command in the shell environment.  Rather than typing the command ''ls -l | less'' repeatedly, a user might create an alias named ''lll'' that when typed yields the same results as if ''ls -l | less'' had been typed.  A shell alias action is an XML ''shell-alias'' element with the following attributes:

^Attribute^Value^
|''shell''|shell type for which this alias is valid|
|''development-env''|set to boolean ''true'' to restrict this alias to only be set on ''vpkg_devrequire''|

The text inside the XML ''shell-alias'' element is the command to be executed for the alias.  A blank command implies that the named alias should be removed from the environment if it exists.  Multiple ''shell-alias'' elements are allowed. 

Example:
<code xml>
<shell-alias name="lll">/bin/ls -l | /usr/bin/less</shell-alias>
<!-- Remove the 'll' alias: -->
<shell-alias name="ll"/>
</code>

==== Dependencies & Incompatibilities ====

Dependencies and incompatibilities are XML elements that contain one or more package identifiers, package identifier pattern matches, or predicates that must:

  * dependencies:
    * package id/pattern must be satisfied (present in environment or successfully loads and configures)
    * predicate evaluates to true
  * incompatibilities:
    * package id/pattern must not be configured in the current environment
    * prediate evaluates to false

A ''pre-condition'' is tested //before// any environment changes are made for the sake of the package.  A ''post-condition'' is tested //after// all environment changes have been made for the sake of the package.

Example:
<code xml>
<dependencies>
  <package id="pgi/14"/>
  <package id="fftw/^^3\."/>
</dependencies>
<incompatibilities>
  <predicate id="gaussian-scratch-test" stage="post-condition">
    <variable>GAUSS_SCRDIR</variable>
    <operator>starts-with</operator>
    <value>/home</value>
    <message>Home directories should not be used as scratch for Gaussian.</message>
  </predicate>
</incompatibilities>
</code>

=== Package Id Patterns ===

A package id pattern is a string that is similar in construction to a regular package id but contains a regular expression for the package side, version side, or both sides of the versioned package id.  A regular expression is indicated in either half by prefixing with a carat (^) character.  For example, the pattern

> ''openmpi/^^1\.[0-9]*[02468]\.''

includes a regular expression in the version half of the id.  Discarding the leading carat, the regular expression is

> ''^1\.[0-9]*[02468]\.''

This pattern will match with any even-numbered release of the ''openmpi'' package in the 1.x series:  1.4.4, 1.6.3, 1.8.2 would all be matches, but 1.5.1 would not.

As a dependency, id patterns are tested against packages that have already been loaded into the environment.  If no matching package has been loaded, VALET locates an available package that matches the pattern and attempts to load it into the environment.  Note that VALET simply chooses the first versioned package it finds as a match, which may or may not be a new enough version, etc.  Ideally, the user will have loaded an appropriate version of the package prior to loading the package that uses an id pattern dependency.

<note tip>Using a regular expression that is the negation of what you would use for a dependency and employing an incompatibility forces the user to load the dependency prior to loading this package.  After all, package ids and id patterns inside an incompatibilities element do not cause VALET to attempt to load a package into the environment.</note>

=== Predicates ===

A predicate is a test performed against an environment variable's value to determine whether package configuration should succeed or fail.  Minimally, an XML ''predicate'' element must contain an environment variable name and an //operator// that specifies what test is to be performed.  Most operators are binary and thus require a value against which to test; the ''is-set'' and ''not-is-set'' operators are unary and take no additional test value.

^Attribute^Value^Optional^
|''operator''|''is-set'': has a value|required|
|:::|''is-not-set'': does not have a value|:::|
|:::|''eq'', ''=='': lexigraphically equivalent to ''value''|:::|
|:::|''ne'', ''!='': not lexigraphically equivalent to ''value''|:::|
|:::|''lt'', ''%%<%%'': lexigraphically less-than ''value''|:::|
|:::|''le'', ''%%<=%%'': lexigraphically less-than or equivalent to ''value''|:::|
|:::|''gt'', ''%%>%%'': lexigraphically greater-than ''value''|:::|
|:::|''ge'', ''%%>=%%'': lexigraphically greater-than or equivalent to ''value''|:::|
|:::|''starts-with'', ''%%<<%%'': ''value'' as prefix|:::|
|:::|''not-starts-with'', ''!%%<<%%'': not ''value'' as prefix|:::|
|:::|''ends-with'', ''%%>>%%'': ''value'' as suffix|:::|
|:::|''not-ends-with'', ''!%%>>%%'': not ''value'' as suffix|:::|
|:::|''contains'', ''%%<>%%'': contains ''value''|:::|
|:::|''not-contains'', ''!%%<>%%'': does not contain ''value''|:::|
|:::|''matches'', ''~'': matches regular expression in ''value''|:::|
|:::|''not-matches'', ''!~'': does not match regular expression in ''value''|:::|
|''stage''|''pre-condition''|default = ''pre-condition''|
|:::|''post-condition''|:::|

The ''predicate'' element may contain the following child elements:

^Element Tag^Purpose^ Optional^
|''variable''|Environment variable name|required|
|''operator''|The attribute can instead be specified as a child element||
|''value''|Value to compare against for binary operators|required except for operator = ''is-set'' or ''not-is-set''|
|''message''|String that will be displayed if the predicate is not satisfied||

Example:
<code xml>
<predicate id="gaussian-scratch-test" stage="post-condition">
  <variable>GAUSS_SCRDIR</variable>
  <operator>starts-with</operator>
  <value>/home</value>
  <message>Home directories should not be used as scratch for Gaussian.</message>
</predicate>
</code>

----

[[software:valet:03_packagefile|Prev: XML Package Files]] | [[software:valet:04_snapshot|Next: JSON/YAML Package Files]]