Keras Python Virtual Environment

This page documents the creation of a Python virtual environment (virtualenv) containing the Keras deep-learning suite on the Caviness HPC system. It assumes that the user is adding the software to the workgroup storage.

Prepare to add software in the standard sub-directories of the workgroup storage:

[user@login01 ~]$ workgroup -g my_workgroup
[(my_workgroup:user)@login01 ~]$ mkdir --mode=2775 --parent ${WORKDIR}/sw/keras
[(my_workgroup:user)@login01 ~]$ mkdir --mode=2775 --parent ${WORKDIR}/sw/valet

These commands create any missing directories. All directories created will have group-write and -inherit permissions.

All versions of the Keras virtualenv will be stored in the common base directory, $WORKDIR/sw/keras; each virtualenv must have a unique name that will become the VALET version of Keras. In this tutorial, the latest version of Keras as of today (2020-02-04) will be installed and the SKLearn package will be added, as well. An appropriate version would be <date>:sklearn, or 20200204:sklearn. That version can be translated to a VALET-friendly directory name:

[(my_workgroup:user)@login01 ~]$ vpkg_id2path --version-id=20200204:sklearn
20200204-sklearn
[(my_workgroup:user)@login01 ~]$ mkdir --mode=3750 ${WORKDIR}/sw/keras/20200204-sklearn

The Intel Python distribution will form the basis for the Keras virtualenv, so add it to the environment:

[(my_workgroup:user)@login01 ~]$ vpkg_require intel-python/2019u5:python3
Adding package `intel-python/2019u5:python3` to your environment
(root) [(my_workgroup:user)@login01 ~]$

Notice the prompt changed: the text (root) now prefixes it, indicating the directory that contains the active Python virtualenv.

The Python environment will include Keras, numpy, and pip:

(root) [(my_workgroup:user)@login01 ~]$ conda create --prefix=${WORKDIR}/sw/keras/20200204-sklearn keras numpy pip

The conda environment builder will take some time to locate all the necessary packages and their dependencies. It will then ask for confirmation:

Proceed ([y]/n)? y

Answer "y" to create the virtualenv. If successful, text will be displayed that mentions source'ing a file to activate the virtualenv: refrain from doing that, and instead use VALET to manage the Keras virtualenv instances. Rollback the intel-python environment changes before proceeding:

(root) [(my_workgroup:user)@login01 ~]$ vpkg_rollback
[(my_workgroup:user)@login01 ~]$ 

Notice the (root) has disappeared from the prompt, indicating that the baseline virtualenv has been deactivated.

Assuming the workgroup does not already have a Keras VALET package definition, the following text:

keras:
    prefix: /work/my_workgroup/sw/keras
    description: KERAS Python environments
    flags:
        - no-standard-paths
    actions:
        - action: source
          script:
              sh: intel-python.sh
          order: failure-first
          success: 0
    versions:
        "20200204:sklearn":
            description: 2020-02-04 conda build with SKLearn added
            dependencies:
                - intel-python/2019u5:python3

would be added to ${WORKDIR}/sw/valet/keras.vpkg_yaml. If that file already exists, add your new version at the same level as others:

keras:
    prefix: /work/my_workgroup/sw/keras
    description: KERAS Python environments
    flags:
        - no-standard-paths
    actions:
        - action: source
          script:
              sh: intel-python.sh
          order: failure-first
          success: 0
    versions:
        "20200204:sklearn":
            description: 2020-02-04 conda build with SKLearn added
            dependencies:
                - intel-python/2019u5:python3
        "20191218":
            description: 2019-12-18 conda build
            dependencies:
                - intel-python/2019u2:python3
        "20191112":
            description: 2019-11-12 conda build
            dependencies:
                - intel-python/2019u5:python3

The SKLearn package is not present in the conda online repositories, but it can be installed using pip. First, activate the new Keras virtualenv:

[(my_workgroup:user)@login01 ~]$ vpkg_require keras/20200204:sklearn
Adding dependency `intel-python/2019u5:python3` to your environment
Adding package `keras/20200204:sklearn` to your environment
(/work/my_workgroup/sw/keras/20200204-sklearn) [(my_workgroup:user)@login01 keras]$ which pip
/work/my_workgroup/sw/keras/20200204-sklearn/bin/pip

A prefix has reappeared on the prompt — the path at which the new Keras virtualenv was created — and the pip command refers to an executable within that directory tree. The virtualenv is ready to have SKLearn installed.

(/work/my_workgroup/sw/keras/20200204-sklearn) [(my_workgroup:user)@login01 ~]$ pip install sklearn
Collecting sklearn
  Downloading ..
    :
Successfully built sklearn
Installing collected packages: joblib, scikit-learn, sklearn
Successfully installed joblib-0.14.1 scikit-learn-0.22.1 sklearn-0.0
(/work/my_workgroup/sw/keras/20200204-sklearn) [(my_workgroup:user)@login01 ~]$ python3 -c "import sklearn;print(sklearn.__version__)"
0.22.1

The Keras environment with SKLearn is now ready for use.

Any job scripts you submit that want to run scripts using this virtualenv should include something like the following toward its end:

#
# Setup Keras virtualenv:
#
vpkg_require keras/20200204:sklearn

#
# Run a Python script in that virtualenv:
#
python3 my_keras_work.py
rc=$?

#
# Do cleanup work, etc....
#

#
# Exit with whatever exit code our Python script handed back:
#
exit $rc