Keras Python Virtual Environment

This page documents the creation of a Python virtual environment (virtualenv) containing the Keras deep-learning suite on the Caviness HPC system. It assumes that the user is adding the software to the workgroup storage.

Prepare to add software in the standard sub-directories of the workgroup storage:

[user@login01 ~]$ workgroup -g my_workgroup
[(my_workgroup:user)@login01 ~]$ mkdir --mode=2775 --parent ${WORKDIR}/sw/keras
[(my_workgroup:user)@login01 ~]$ mkdir --mode=2775 --parent ${WORKDIR}/sw/valet

These commands create any missing directories. All directories created will have group-write and -inherit permissions.

All versions of the Keras virtualenv will be stored in the common base directory, $WORKDIR/sw/keras; each virtualenv must have a unique name that will become the VALET version of Keras. In this tutorial, the latest version of Keras as of today (2020-02-04) will be installed and the SKLearn package will be added, as well. An appropriate version would be <date>:sklearn, or 20200204:sklearn. That version can be translated to a VALET-friendly directory name:

[(my_workgroup:user)@login01 ~]$ vpkg_id2path --version-id=20200204:sklearn
20200204-sklearn
[(my_workgroup:user)@login01 ~]$ mkdir --mode=3750 ${WORKDIR}/sw/keras/20200204-sklearn

The Intel Python distribution will form the basis for the Keras virtualenv, so add it to the environment:

[(my_workgroup:user)@login01 ~]$ vpkg_require intel-python/2019u5:python3
Adding package `intel-python/2019u5:python3` to your environment
(root) [(it_nss:frey)@login01 ~]$

Notice the prompt changed: the text (root) now prefixes it, indicating the directory that contains the active Python virtualenv.

The Python environment will include Keras, numpy, and pip:

(root) [(it_nss:frey)@login01 keras]$ conda create --prefix=${WORKDIR}/sw/keras/20200204-sklearn keras numpy pip

The conda environment builder will take some time to locate all the necessary packages and their dependencies. It will then ask for confirmation:

Proceed ([y]/n)? y

Answer "y" to create the virtualenv. If successful, text will be displayed that mentions source'ing a file to activate the virtualenv: refrain from doing that, and instead use VALET to manage the Keras virtualenv instances.

Assuming the workgroup does not already have a Keras VALET package definition, the following text:

keras:
    prefix: /work/my_workgroup/sw/keras
    description: KERAS Python environments
    actions:
        - action: source
          script:
              sh: anaconda-activate.sh
          order: failure-first
          success: 0
    versions:
        "20200204:sklearn":
            description: 2020-02-04 conda build with SKLearn added
            dependencies:
                - intel-python/2019u5:python3

would be added to ${WORKDIR}/sw/valet/keras.vpkg_yaml. If the file already exists, add your new version at the same level as others:

keras:
    prefix: /work/my_workgroup/sw/keras
    description: KERAS Python environments
    actions:
        - action: source
          script:
              sh: anaconda-activate.sh
          order: failure-first
          success: 0
    versions:
        "20200204:sklearn":
            description: 2020-02-04 conda build with SKLearn added
            dependencies:
                - intel-python/2019u5:python3
        "20191218":
            description: 2019-12-18 conda build
            dependencies:
                - intel-python/2019u2:python3
        "20191112":
            description: 2019-11-12 conda build
            dependencies:
                - intel-python/2019u5:python3