JSON Package Definition Files

In version 2 of VALET, packages (and their versions) can also defined in JSON files. JSON – short for JavaScript Object Notation – is an open standard format that uses human-readable text to represent structured data, just as XML does. XML carries with it a heavy amount of additional formatting surrounding the data, which some people find makes the data itself less visible when inspected. JSON uses far less formatting around the data.

VALET package definitions in the JSON format use the file extension .vpkg_json. Comments can be included in the files using the hash character:

#
# This is a comment.
#
{
  "matlab": {
  }
}

The JSON package file consists minimally of a dictionary containing a single key-value pair:

{
  "<<pkg id>>": {
  }
}

The «pkg id» must match with the name of the file itself. For example, mathematica.vpkg_json would contain:

{
  "mathematica": {
    "description":   "Wolfram Mathematica",
    "url":           "http://www.wolfram.com/",
    "prefix":        "/opt/mathematica"
  }
}

A package can optionally have a human-readable description and a reference URL (e.g. the vendor/product web site) as shown in this example.

Ideally, all versions of a package are installed in unique directories under a common parent directory. If this is indeed the case, then a prefix key-value pair can be used as shown above. When the individual package versions are later defined, they can provide a relative path in their own prefix key-value pair which VALET takes as being relative to the package's prefix path.

All versions of a package are contained within a dictionary keyed by the string versions:

{
  "mathematica": {
    "description":         "Wolfram Mathematica",
    "url":                 "http://www.wolfram.com/",
    "prefix":              "/opt/mathematica",
 
    "versions": {
      "6": {
        "description":     "Wolfram Mathematica version 6",
        "url":             "http://www.wolfram.com/6",
        "prefix":          "6.0.1",
 
        "dependencies": [
          "pgi/10"
        ],
 
        "incompatibilities": [
          "matlab/^^.*$"
        ],
 
        "actions": [
          {
            "bindir":        "Executables"
          }
        ]
      }
    }
  }
}

The optional description and url key-value pairs override the value presented in the parent package element.

As mentioned earlier, the prefix value specified here is a relative path that VALET assumes is relative to the prefix provided by the parent package. In this case /opt/mathematica/6.0.1 would be the full path.

The actions key-value pair defines the changes that accompany package inclusion. The simplest actions involve the addition of filesystem paths to various environment variables. In this example, the bindir action specifies a path that should be added to the PATH environment variable. Relative paths are taken to be relative to the prefix directory. Besides bindir there are several other special directory-related actions:

By default VALET will also automatically check for the default directory(s) associated with each of the special actions listed above in addition to any that are specified explicitly (this behavior can be disabled on a per-package and per-version basis). In all cases, only directories that actually exist will be setup in the environment.

For software packages that are built using other packages managed by VALET, a list of dependencies can be specified. Each item in the depencencies array can be:

1. A versioned package id, e.g. openmpi/1.8.2
2. A versioned package regular expression, e.g. openmpi/^^1\\.[0-9]*[02468]\\.
3. A predicate

The regular expression format and predicates will be covered later in this document. When configuring the environment, VALET will automatically attempt to include all dependency packages' configuration details, as well.

A list of incompatibilities can be included to express that a versioned package conflicts with some other versioned package. Just like dependencies, the incompatibilities array can contain

1. Versioned package ids, e.g. openmpi/1.8.2
2. Versioned package regular expressions, e.g. openmpi/^^1\\.[0-9]*[02468]\\.
3. Predicates

In the example above Mathematica 6 conflicts (somehow) with any version of Matlab: the ^^.*$ regular expression matches any version id associated with the matlab package. Inclusion of any version of Matlab either prior to or after this versioned package's inclusion will yield an error.

Multiple version dictionaries (keyed by their version id) may be present in a package's versions dictionary. A default-version element in the package indicates which version should be considered the default; omitting the default-version element, the first version dictionary will be the default for the package (the { … } has been used to shorten the lengthy version dictionaries for illustrative purposes):

{
  "mathematica": {
    "description":         "Wolfram Mathematica",
    "url":                 "http://www.wolfram.com/",
    "prefix":              "/opt/mathematica",
 
    "default-version":     "7",
 
    "versions": {
      "6": { ... },
      "7": { ... },
      "8": { ... }
    }
  }
}

A version alias is a unique package version that points to a sibling version. For example, Gaussian releases their G09 product and its incremental updates as a series, e.g. g09a01, g09a02, g09b03, etc. Most users probably don't care what revision of G09 they are using, so long as they are using G09. By specifying a version alias

{
  "gaussian": {
    "versions": {
      "g09a01": { ... },
      "g09a02": { ... },
      "g09b03": { ... },
 
      "g09": { "alias-to": "g09b03" }
    }
  }
}

users can always just request gaussian/g09 and the system administrator can change the alias target as future incremental updates are installed on the system. As illustrated, a version alias dictionary has no content other than the alias-to key-value pair.

In some cases there may be additional environment variables that need to be modified for a package/version. To facilitate this, zero or more variable actions can be added to a package or version dictionary:

{
  "mathematica": {
    "description":         "Wolfram Mathematica",
    "url":                 "http://www.wolfram.com/",
    "prefix":              "/opt/mathematica",
 
    "default-version":     "7",
 
    "actions": [
      { "variable": "MATHEMATICA_IN_USE", "value": 1 }
    ],
 
    "versions": {
      "6": { ... },
      "7": { ... },
      "8": {
        ...
 
        "actions": [
          { "variable": "MATHEMATICA_DOCS", "action": "path-prepend", "value": "${HOME}/mma_8" }
        ]
      }
    }
  }
}

The action can be:

If no action attribute is specified the set action is assumed.

VALET defines some environment variables that variable actions may reference:

If there is additional "work" that cannot be encapsulated in VALET configuration directives, the system administrator can write an external script that will be sourced into the user's shell. A script action provides the path to each such script and directives that control how that path is used:

    ...
        "actions": [
          { "variable": "MATHEMATICA_DOCS", "action": "path-prepend", "value": "${HOME}/mma_8" },
          {
            "action": "source",
            "script": {
              "sh": "/opt/mathematica/bin/remote_kernel.sh",
              "csh": "/opt/mathematica/bin/remote_kernel.csh"
            },
            "order": "failure-first",
            "success": 0
          }
        ] 
    ...

The same VALET-defined variables available to variable actions can be used by scripts (e.g. using VALET_PATH_PREFIX to find the software's installed location). Scripts (or binary executables) can be forked-and-executed ("action": "exec") while scripts can be sourced into the shell, as in the example above. Shell-specific paths can be specified. An integer value for success and/or failure can be provided that will be compared against the return code after source'ing/execution of the appropriate path. The order determines which is tested first: failure-first or success-first. A script action that fails its succeed/failure test causes VALET to discard the current configuration changes and return in error.

In some cases it may be necessary to test the value of one or more environment variables to determine whether or not the package setup should succeed or fail. For example, if the user has no GAUSS_SCRDIR defined then Gaussian will have no directory in which to write scratch files. This could be described as either a dependency or an incompatibility.

Such tests are modeled with a predicate. A predicate can be specified inside the dependencies and incompatibilities dictionaries of a package or a version. A predicate is defined similarly to a variable action:

  ...
    "dependencies": [
      { "variable": "GAUSS_SCRDIR", "operator": "is-set", "stage": "post-condition" }
    }   
  ...

In this case, the predicate could also be configured as an incompatibility:

  ...
    "incompatibilities": [
      { "variable": "GAUSS_SCRDIR", "operator": "not-is-set", "stage": "post-condition" }
    }
  ...

The stage key-value pair dictates whether the test should be performed before (pre-condition) or after (post-condition) VALET makes changes to the environment on behalf of the package. If no stage attribute is specified, the predicate is a pre-condition.

Valid operators are:

A set of file-testing operators exist, as well. For these operators, the variable key may be replaced by the path key in the predicate specification. Tests include existence/non-existence of a filesystem entity at the path; read/write/execute access checks; and file type introspection (directory, regular file, socket, etc.).

For binary operators, the required argument is provided in a value key-value pair. For example, to ensure that setup does not proceed if a user has pre-set GAUSS_SCRDIR to a filesystem that should not be used as scratch, the following incompatibility predicate might be used:

  ...
    "incompatibilities": [
      { "variable": "GAUSS_SCRDIR", "operator": "matches", "value": "^/(home|archive)/",
        "message": "Storing Gaussian scratch files on /home or /archive is forbidden."
      }
    }
  ...

If a user subsequently attempts to vpkg_require gaussian the message will be displayed if they have GAUSS_SCRDIR misconfigured:

ERROR: an error occurred while altering your environment
REASON: Storing Gaussian scratch files on /home or /archive is forbidden.

All predicates must pass in order for environment configuration to succeed; essentially, success is the logical AND of all predicates specified.

A VALET package is represented in JSON as a dictionary containing a single key-value pair:

Example:

{
  "gaussian": {
       :
     "versions": {
       :
     }
   }
}

A set of scalar properties can be used both in the global and per-version areas of the configuration. Scalar properties are:

Typically, the package-global prefix is set to a directory containing each version of the package, and each versioned package defines a prefix relative to that directory:

$ ls -l /opt/shared/openmpi
total 14
drwxr-sr-x 8 user   sysadmin 8 Aug 27 16:10 1.8.2
drwxr-sr-x 8 user   sysadmin 8 Aug 27 16:10 1.8.2-gcc-4.8.3
drwxr-sr-x 8 user   sysadmin 8 Aug 28 09:26 1.8.2-intel64
drwxrwsr-x 2 user   sysadmin 3 Aug 27 17:10 attic

The package's prefix = /opt/shared/openmpi and versions would use e.g. prefix = 1.8.2-intel64. If no prefix is provided for a versioned package, then VALET implicitly uses the version id.

Versioned packages that do not specify the development-env or standard-paths flag inherit the value of their parent package.

Example:

{
  "gaussian": {
    "description": "Gaussian Quantum Chemistry Suite",
    "url": "http://gaussian.com/",
    "prefix": "/opt/shared/gaussian",
 
    "development-env": false,
    "standard-paths": false,
 
    "versions": {
      "g09": {
        "prefix": "g09d01"
      },
      "g09d01": {
      }
    }
  }
}

In this example, version g09d01 has a prefix path of /opt/shared/gaussian/g09d01 since it did not specify a prefix. Version g09 provided a relative path that is appended to the prefix of its parent package to also yield /opt/shared/gaussian/g09d01. Both versions inherit the development-env and standard-paths behavior of the parent package.

The full versioned package identifiers available in the gaussian package defined above would be:

• gaussian/g09
• gaussian/g09d01

Since the g09 version targets the same executables as g09d01, the g09 version can be considered to be an alias of g09d01. VALET allows for aliased versions of packages:

"versions": {
  "g09": { "alias-to": "g09d01" },
  "g09d01": {
  }
}

Once the overall versioning structure of a package has been created, the environment tests and changes associated with the package and versions of the package can be added. The three kinds of tests/changes are:

All three are represented as JSON arrays. If present in the package-global area of the configuration, the tests/changes affect all versions of the package and are satisfied/applied before version-specific dependencies/incompatibilities and actions.

Example:

{
  "gaussian": {
    "description": "Gaussian Quantum Chemistry Suite",
    "url": "http://gaussian.com/",
    "prefix": "/opt/shared/gaussian",
 
    "development-env": false,
    "standard-paths": false,
 
    "dependencies": [
      { "variable": "GAUSS_SCRDIR", "operator": "not-is-set", "stage": "pre-condition",
        "message": "If GAUSS_SCRDIR is not set, the working directory will be used; you do not want that."
      }
    ],
 
    "actions": [
      { "action": "exec", "script": { "all": "mk-gaussian-scrdir" },
        "order": "success-first", "success": 0
      }
    ],
 
    "versions": {
      "g09": { "alias-to": "g09d01" },
      "g09d01": {
        "dependencies": [
          "pgi/14"
        ],
        "actions": [
          { "variable": "GAUSSIAN_VERSION", "action": "set", "value": "G09" },
          { "action": "source", "script": { "sh": "g09.sh" } }
        ]
      }
    }
  }
}

Given the package definition above, typing vpkg_require gaussian/g09 yields the following course of events:

1. g09 is an alias to g09d01, so version g09d01 becomes the target versioned package being configured
2. Has the pgi/14 package been loaded into the environment yet?
   • If not, do all of its tests and attempt to affect all of its environment changes before proceeding
     • If that fails, this configuration fails and must exit immediately
3. The installation prefix for the software is constructed as /opt/shared/gaussian/g09d01
4. The GAUSS_SCRDIR environment variable is examined
   • If it does not exist in the environment the configuration fails and the message text is printed to stdout
5. The mk-gaussian-scrdir program found in VALET's libexec directory is executed
   • If the result code returned by mk-gaussian-scrdir is zero, continue
   • Otherwise, the configuration fails and exits
6. The GAUSSIAN_VERSION environment variable is assigned the value G09
7. For Bash and other sh-like shells, the script g09.sh in VALET's libexec directory is sourced
   • The return code is not checked for success or failure

If control reaches the end of this pseudo-code, then the gaussian/g09 package has been successfully configured in the environment.

An actions array can be specified either in a package or in a version of a package and contains zero or more action objects:

• variable – make changes to environment variables
• special directories – shorthand for alterations to PATH, LD_LIBRARY_PATH, etc.
• script – source/execute a script
• shell alias – set/unset an alias

Each action is represented as a JSON dictionary . Actions attached to the package itself will be performed when any version of the package is selected for addition to the environment, and in addition to actions associated with the version itself.

Actions will be performed in the order they are specified in the array.

Example:

"actions": [
  {
    "incdir": "/opt/include"
  },
  { "variable": "PATH", "action": "scrub-path", "value": "/opt/bin" },
  {
    "bindir": "/opt/bin",
    "libdir": [ "/opt/lib", "/opt/lib64" ],
    "mandir": "/opt/share/man"
  },
  { "variable": "USER_SW_PATH", "value": "/opt/bin" },
  { "variable": "CFLAGS", "action": "append-space", "value": "-O3 -g", "development-env": true }
]

Variable actions alter the value of an environment variable. A variable action is a JSON dictionary with the following key-value pairs:

The actions are self-explanatory for the most part. The scrub action removes all occurrences of value from the variables current value in the environment. The scrub-path action treats the value of variable as a search path and removes from that search path all occurrences (by exact match) of value.

The value may contain references to other environment variables using the syntax ${ENV VAR NAME}.

Example:

{ "variable": "PATH", "action": "scrub-path", "value": "/opt/bin" }

All special-directory actions are enclosed in a JSON dictionary within the actions array. These actions are shortcuts that produce variable actions that affect standard environment variables (like PATH or LD_LIBRARY_PATH). The dictionary can contain the following key-value pairs; for each, the value can be either a single string or an array of strings:

Example:

{
  "bindir": "/opt/bin",
  "libdir": [ "/opt/lib", "/opt/lib64" ],
  "mandir": "/opt/share/man"
}

At times VALET may not be able to fully express the modifications necessary for a package, or the package includes its own extensive environment configuration scripts. In such instances, a VALET package can be configured to simply execute (or source) that external script. A script action is a JSON dictionary containing the following key-value pairs:

The script dictionary provides alternative paths for each different shell supported by VALET.

The order determines which is tested first, the success return code or the failure return code. If no success or failure are specified, then the return code is not tested. Many shell scripts return code 0 on success and a non-zero value in case of a failure; this could be tested by supplying order = success_first and success = 0.

Example:

{
  "action": "source",
  "script": {
    "sh": "/opt/mathematica/bin/remote_kernel.sh",
    "csh": "/opt/mathematica/bin/remote_kernel.csh"
  },
  "order": "success-first",
  "success": 0
}

An alias is a word that represents an arbitrarily complex command in the shell environment. Rather than typing the command ls -l | less repeatedly, a user might create an alias named lll that when typed yields the same results as if ls -l | less had been typed. A shell alias action is a JSON dictionary containing the following key-value pairs:

The command dictionary provides alternative commands for each different shell supported by VALET. For simple commands that do not use shell-specific syntax, the shell id any (or *) can be used. A blank command implies that the named alias should be removed from the environment if it exists.

Example:

{
  # Add 'lll' alias:
  "shell-alias": "lll",
  "command": { "any": "/bin/ls -l | /usr/bin/less" }
},
{
  # Remove 'll' alias:
  "shell-alias": "ll",
  "command": { "*": "" }
}

A warning message is quite simply that: a textual message that is written to stdout when the user configures the package into his/her environment. A warning might be used to notify the user that s/he is using an unsupported version of a product, for example. A warning message action is a JSON dictionary containing the following key-value pairs:

Example:

{
  # Mention that we don't support PGI on this cluster:
  "warning": "The Portland compiler suite is not officially supported on this cluster."
}

Dependencies and incompatibilities are JSON arrays of package identifiers, package identifier pattern matches, or predicates that must:

• dependencies:
  • package id/pattern must be satisfied (present in environment or successfully loads and configures)
  • predicate evaluates to true
• incompatibilities:
  • package id/pattern must not be configured in the current environment
  • prediate evaluates to false

A pre-condition is tested before any environment changes are made for the sake of the package. A post-condition is tested after all environment changes have been made for the sake of the package.

Examples:

"dependencies": [
  "pgi/14",
  "fftw/^^3\\."
],
"incompatibilities": [
  {
    "variable": "GAUSS_SCRDIR",
    "operator": "starts-with",
    "value":    "/home",
    "stage":    "post-condition",
    "message":  "Home directories should not be used as scratch for Gaussian."
  }
]

A package id pattern is a string that is similar in construction to a regular package id but contains a regular expression for the package side, version side, or both sides of the versioned package id. A regular expression is indicated in either half by prefixing with a carat (^) character. For example, the pattern

"openmpi/^^1\\.[0-9]*[02468]\\."

includes a regular expression in the version half of the id. Discarding the leading carat, the regular expression is

^1\\.[0-9]*[02468]\\.

This pattern will match with any even-numbered release of the openmpi package in the 1.x series: 1.4.4, 1.6.3, 1.8.2 would all be matches, but 1.5.1 would not. JSON treats the backslash as an escape character, hence the reason it must be escaped itself in the regular expression string.

As a dependency, id patterns are tested against packages that have already been loaded into the environment. If no matching package has been loaded, VALET locates an available package that matches the pattern and attempts to load it into the environment. Note that VALET simply chooses the first versioned package it finds as a match, which may or may not be a new enough version, etc. Ideally, the user will have loaded an appropriate version of the package prior to loading the package that uses an id pattern dependency.

A predicate is a test performed against an environment variable's value to determine whether package configuration should succeed or fail. Minimally, a predicate dictionary must contain an environment variable name and an operator that specifies what test is to be performed. Most operators are binary and thus require a value against which to test; the is-set, not-is-set, exists, not-exists, is-readable, not-is-readable, is-writable, not-is-writable, is-executable, and not-is-executable operators are unary and take no additional test value.

There are several predicate operators used to examine filesystem paths. The is-file-type operator uses the stat() system call on the given path; if the path is a symbolic link, then the type refers to the target of the link, not the link itself. Use the is-strict-file-type operator to not follow symbolic links. The right-hand-side of this kind of predicate (its value) should be one of the following strings:

Examples:

{
  "variable": "GAUSS_SCRDIR",
  "operator": "starts-with",
  "value":    "/home",
  "stage":    "post-condition",
  "message":  "Home directories should not be used as scratch for Gaussian."
},
{
  "path":     "/tmp/.pgSQL",
  "operator": "is-file-type",
  "value":    "socket",
  "message":  "Gaussian database access expects a PostgreSQL socket at /tmp/.pgSQL for connections."
},
{
  "path":     "/opt/shared/gaussian",
  "operator": "is-readable",
  "message":  "Gaussian software is only licensed to UD users."
}