hpc documentation - technical:recipes:keras-in-virtualenv

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Keras Python Virtual Environment ======

This page documents the creation of a Python virtual environment (virtualenv) containing the Keras deep-learning suite on the Caviness HPC system.  It assumes that the user is adding the software to the workgroup storage.

===== Prepare Workgroup Directory =====

Prepare to add software in the standard sub-directories of the workgroup storage:

<code bash>
[user@login01 ~]$ workgroup -g my_workgroup
[(my_workgroup:user)@login01 ~]$ mkdir --mode=2775 --parent ${WORKDIR}/sw/keras
[(my_workgroup:user)@login01 ~]$ mkdir --mode=2775 --parent ${WORKDIR}/sw/valet
</code>

These commands create any missing directories.  All directories created will have group-write and -inherit permissions.

===== Create Keras Virtualenv =====

All versions of the Keras virtualenv will be stored in the common base directory, ''$WORKDIR/sw/keras''; each virtualenv must have a unique name that will become the VALET version of Keras.  In this tutorial, the latest version of Keras as of today (2020-02-04) will be installed and the SKLearn package will be added, as well.  An appropriate version would be ''<date>:sklearn'', or ''20200204:sklearn''.  That version can be translated to a VALET-friendly directory name:

<code bash>
[(my_workgroup:user)@login01 ~]$ vpkg_id2path --version-id=20200204:sklearn
20200204-sklearn
[(my_workgroup:user)@login01 ~]$ mkdir --mode=3750 ${WORKDIR}/sw/keras/20200204-sklearn
</code>

The Intel Python distribution will form the basis for the Keras virtualenv, so add it to the environment:

<code bash>
[(my_workgroup:user)@login01 ~]$ vpkg_require intel-python/2019u5:python3
Adding package `intel-python/2019u5:python3` to your environment
(root) [(my_workgroup:user)@login01 ~]$
</code>

Notice the prompt changed:  the text ''(root)'' now prefixes it, indicating the directory that contains the active Python virtualenv.

The Python environment will include Keras, numpy, and pip:

<code bash>
(root) [(my_workgroup:user)@login01 ~]$ conda create --prefix=${WORKDIR}/sw/keras/20200204-sklearn keras numpy pip
</code>

The conda environment builder will take some time to locate all the necessary packages and their dependencies.  It will then ask for confirmation:

<code bash>
Proceed ([y]/n)? y
</code>

Answer "y" to create the virtualenv.  If successful, text will be displayed that mentions source'ing a file to activate the virtualenv:  refrain from doing that, and instead use VALET to manage the Keras virtualenv instances.  Rollback the ''intel-python'' environment changes before proceeding:

<code bash>
(root) [(my_workgroup:user)@login01 ~]$ vpkg_rollback
[(my_workgroup:user)@login01 ~]$ 
</code>

Notice the ''(root)'' has disappeared from the prompt, indicating that the baseline virtualenv has been deactivated.
===== VALET Package Definition =====

Assuming the workgroup does //not// already have a Keras VALET package definition, the following text:

<file keras.vpkg_yaml>
keras:
    prefix: /work/my_workgroup/sw/keras
    description: KERAS Python environments
    flags:
        - no-standard-paths
    actions:
        - action: source
          script:
              sh: intel-python.sh
          order: failure-first
          success: 0
    versions:
        "20200204:sklearn":
            description: 2020-02-04 conda build with SKLearn added
            dependencies:
                - intel-python/2019u5:python3
</file>

would be added to ''${WORKDIR}/sw/valet/keras.vpkg_yaml''.  If that file already exists, add your new version at the same level as others:

<file keras.vpkg_yaml>
keras:
    prefix: /work/my_workgroup/sw/keras
    description: KERAS Python environments
    flags:
        - no-standard-paths
    actions:
        - action: source
          script:
              sh: intel-python.sh
          order: failure-first
          success: 0
    versions:
        "20200204:sklearn":
            description: 2020-02-04 conda build with SKLearn added
            dependencies:
                - intel-python/2019u5:python3
        "20191218":
            description: 2019-12-18 conda build
            dependencies:
                - intel-python/2019u2:python3
        "20191112":
            description: 2019-11-12 conda build
            dependencies:
                - intel-python/2019u5:python3
</file>

<note tip>On Caviness after a user has used the ''workgroup'' command, VALET searches for package definitions in ''${WORKDIR}/sw/valet'' by default.</note>

===== Install SKLearn =====

The SKLearn package is not present in the conda online repositories, but it can be installed using ''pip.''  First, activate the new Keras virtualenv:

<code bash>
[(my_workgroup:user)@login01 ~]$ vpkg_require keras/20200204:sklearn
Adding dependency `intel-python/2019u5:python3` to your environment
Adding package `keras/20200204:sklearn` to your environment
(/work/my_workgroup/sw/keras/20200204-sklearn) [(my_workgroup:user)@login01 keras]$ which pip
/work/my_workgroup/sw/keras/20200204-sklearn/bin/pip
</code>

A prefix has reappeared on the prompt — the path at which the new Keras virtualenv was created — and the ''pip'' command refers to an executable within that directory tree.  The virtualenv is ready to have SKLearn installed.

<code bash>
(/work/my_workgroup/sw/keras/20200204-sklearn) [(my_workgroup:user)@login01 ~]$ pip install sklearn
Collecting sklearn
  Downloading ..
    :
Successfully built sklearn
Installing collected packages: joblib, scikit-learn, sklearn
Successfully installed joblib-0.14.1 scikit-learn-0.22.1 sklearn-0.0
(/work/my_workgroup/sw/keras/20200204-sklearn) [(my_workgroup:user)@login01 ~]$ python3 -c "import sklearn;print(sklearn.__version__)"
0.22.1
</code>

The Keras environment with SKLearn is now ready for use.

===== Job Scripts =====

Any job scripts you submit that want to run scripts using this virtualenv should include something like the following toward its end:

<code>
#
# Setup Keras virtualenv:
#
vpkg_require keras/20200204:sklearn

#
# Run a Python script in that virtualenv:
#
python3 my_keras_work.py
rc=$?

#
# Do cleanup work, etc....
#

#
# Exit with whatever exit code our Python script handed back:
#
exit $rc
</code>