Keras Python Virtual Environment

This page documents the creation of a Python virtual environment (virtualenv) containing the Keras deep-learning suite on the Caviness HPC system. It assumes that the user is adding the software to the workgroup storage.

Prepare to add software in the standard sub-directories of the workgroup storage:

[user@login01 ~]$ workgroup -g my_workgroup
[(my_workgroup:user)@login01 ~]$ mkdir --mode=2775 --parent ${WORKDIR}/sw/keras
[(my_workgroup:user)@login01 ~]$ mkdir --mode=2775 --parent ${WORKDIR}/sw/valet

These commands create any missing directories. All directories created will have group-write and -inherit permissions.

All versions of the Keras virtualenv will be stored in the common base directory, $WORKDIR/sw/keras; each virtualenv must have a unique name that will become the VALET version of Keras. In this tutorial, the latest version of Keras as of today (2020-02-04) will be installed and the SKLearn package will be added, as well. An appropriate version would be <date>:sklearn, or 20200204:sklearn. That version can be translated to a VALET-friendly directory name:

[(my_workgroup:user)@login01 ~]$ vpkg_id2path --version-id=20200204:sklearn
20200204-sklearn
[(my_workgroup:user)@login01 ~]$ mkdir --mode=3750 ${WORKDIR}/sw/keras/20200204-sklearn

The Intel Python distribution will form the basis for the Keras virtualenv, so add it to the environment:

[(my_workgroup:user)@login01 ~]$ vpkg_require intel-python/2019u5:python3
Adding package `intel-python/2019u5:python3` to your environment
(root) [(my_workgroup:user)@login01 ~]$

Notice the prompt changed: the text (root) now prefixes it, indicating the directory that contains the active Python virtualenv.

The Python environment will include Keras, numpy, and pip:

(root) [(my_workgroup:user)@login01 ~]$ conda create --prefix=${WORKDIR}/sw/keras/20200204-sklearn keras numpy pip

The conda environment builder will take some time to locate all the necessary packages and their dependencies. It will then ask for confirmation:

Proceed ([y]/n)? y

Answer "y" to create the virtualenv. If successful, text will be displayed that mentions source'ing a file to activate the virtualenv: refrain from doing that, and instead use VALET to manage the Keras virtualenv instances. Rollback the intel-python environment changes before proceeding:

(root) [(my_workgroup:user)@login01 ~]$ vpkg_rollback
[(my_workgroup:user)@login01 ~]$ 

Notice the (root) has disappeared from the prompt, indicating that the baseline virtualenv has been deactivated.

Assuming the workgroup does not already have a Keras VALET package definition, the following text:

keras:
    prefix: /work/my_workgroup/sw/keras
    description: KERAS Python environments
    actions:
        - action: source
          script:
              sh: anaconda-activate.sh
          order: failure-first
          success: 0
    versions:
        "20200204:sklearn":
            description: 2020-02-04 conda build with SKLearn added
            dependencies:
                - intel-python/2019u5:python3

would be added to ${WORKDIR}/sw/valet/keras.vpkg_yaml. If that file already exists, add your new version at the same level as others:

keras:
    prefix: /work/my_workgroup/sw/keras
    description: KERAS Python environments
    actions:
        - action: source
          script:
              sh: anaconda-activate.sh
          order: failure-first
          success: 0
    versions:
        "20200204:sklearn":
            description: 2020-02-04 conda build with SKLearn added
            dependencies:
                - intel-python/2019u5:python3
        "20191218":
            description: 2019-12-18 conda build
            dependencies:
                - intel-python/2019u2:python3
        "20191112":
            description: 2019-11-12 conda build
            dependencies:
                - intel-python/2019u5:python3

The SKLearn package is not present in the conda online repositories, but it can be installed using pip. First, activate the new Keras virtualenv:

[(my_workgroup:user)@login01 ~]$ vpkg_require keras/20200204:sklearn
Adding dependency `intel-python/2019u5:python3` to your environment
Adding package `keras/20200204:sklearn` to your environment
(/work/my_workgroup/sw/keras/20200204-sklearn) [(my_workgroup:user)@login01 keras]$ which pip
/work/my_workgroup/sw/keras/20200204-sklearn/bin/pip

A prefix has reappeared on the prompt — the path at which the new Keras virtualenv was created — and the pip command refers to an executable within that directory tree. The virtualenv is ready to have SKLearn installed.

(/work/my_workgroup/sw/keras/20200204-sklearn) [(my_workgroup:user)@login01 ~]$ pip install sklearn
Collecting sklearn
  Downloading ..
    :
Successfully built sklearn
Installing collected packages: joblib, scikit-learn, sklearn
Successfully installed joblib-0.14.1 scikit-learn-0.22.1 sklearn-0.0
(/work/my_workgroup/sw/keras/20200204-sklearn) [(my_workgroup:user)@login01 ~]$ python3 -c "import sklearn;print(sklearn.__version__)"
0.22.1

The Keras environment with SKLearn is now ready for use.

Any job scripts you submit that want to run scripts using this virtualenv should include something like the following:

#
# Setup Keras virtualenv:
#
vpkg_require keras/20200204:sklearn

#
# Run a Python script in that virtualenv:
#
python3 my_keras_work.py
rc=$?

#
# Do cleanup work, etc....
#

#
# Exit with whatever exit code our Python script handed back:
#
exit $rc