This is an old revision of the document!
Keras Python Virtual Environment
This page documents the creation of a Python virtual environment (virtualenv) containing the Keras deep-learning suite on the Caviness HPC system. It assumes that the user is adding the software to the workgroup storage.
Prepare Workgroup Directory
Prepare to add software in the standard sub-directories of the workgroup storage:
[user@login01 ~]$ workgroup -g my_workgroup [(my_workgroup:user)@login01 ~]$ mkdir --mode=2775 --parent ${WORKDIR}/sw/keras [(my_workgroup:user)@login01 ~]$ mkdir --mode=2775 --parent ${WORKDIR}/sw/valet
These commands create any missing directories. All directories created will have group-write and -inherit permissions.
Create Keras Virtualenv
All versions of the Keras virtualenv will be stored in the common base directory, $WORKDIR/sw/keras
; each virtualenv must have a unique name that will become the VALET version of Keras. In this tutorial, the latest version of Keras as of today (2020-02-04) will be installed and the SKLearn package will be added, as well. An appropriate version would be <date>:sklearn
, or 20200204:sklearn
. That version can be translated to a VALET-friendly directory name:
[(my_workgroup:user)@login01 ~]$ vpkg_id2path --version-id=20200204:sklearn 20200204-sklearn [(my_workgroup:user)@login01 ~]$ mkdir --mode=3750 ${WORKDIR}/sw/keras/20200204-sklearn
The Intel Python distribution will form the basis for the Keras virtualenv, so add it to the environment:
[(my_workgroup:user)@login01 ~]$ vpkg_require intel-python/2019u5:python3 Adding package `intel-python/2019u5:python3` to your environment (root) [(it_nss:frey)@login01 ~]$
Notice the prompt changed: the text (root)
now prefixes it, indicating the directory that contains the active Python virtualenv.
The Python environment will include Keras, numpy, and pip:
(root) [(it_nss:frey)@login01 ~]$ conda create --prefix=${WORKDIR}/sw/keras/20200204-sklearn keras numpy pip
The conda environment builder will take some time to locate all the necessary packages and their dependencies. It will then ask for confirmation:
Proceed ([y]/n)? y
Answer "y" to create the virtualenv. If successful, text will be displayed that mentions source'ing a file to activate the virtualenv: refrain from doing that, and instead use VALET to manage the Keras virtualenv instances. Rollback the intel-python
environment changes before proceeding:
(root) [(it_nss:frey)@login01 ~]$ vpkg_rollback [(it_nss:frey)@login01 ~]$
Notice the (root)
has disappeared from the prompt, indicating that the baseline virtualenv has been deactivated.
VALET Package Definition
Assuming the workgroup does not already have a Keras VALET package definition, the following text:
keras: prefix: /work/my_workgroup/sw/keras description: KERAS Python environments actions: - action: source script: sh: anaconda-activate.sh order: failure-first success: 0 versions: "20200204:sklearn": description: 2020-02-04 conda build with SKLearn added dependencies: - intel-python/2019u5:python3
would be added to ${WORKDIR}/sw/valet/keras.vpkg_yaml
. If that file already exists, add your new version at the same level as others:
keras: prefix: /work/my_workgroup/sw/keras description: KERAS Python environments actions: - action: source script: sh: anaconda-activate.sh order: failure-first success: 0 versions: "20200204:sklearn": description: 2020-02-04 conda build with SKLearn added dependencies: - intel-python/2019u5:python3 "20191218": description: 2019-12-18 conda build dependencies: - intel-python/2019u2:python3 "20191112": description: 2019-11-12 conda build dependencies: - intel-python/2019u5:python3
workgroup
command, VALET searches for package definitions in ${WORKDIR}/sw/valet
by default.