technical:recipes:keras-in-virtualenv

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		 Previous revision
technical:recipes:keras-in-virtualenv [2020-02-04 12:22] – created freytechnical:recipes:keras-in-virtualenv [2021-08-06 13:00] (current) – frey
Line 30: Line 30:
 [(my_workgroup:user)@login01 ~]$ vpkg_require intel-python/2019u5:python3 [(my_workgroup:user)@login01 ~]$ vpkg_require intel-python/2019u5:python3
 Adding package `intel-python/2019u5:python3` to your environment Adding package `intel-python/2019u5:python3` to your environment
-(root) [(it_nss:frey)@login01 ~]$+(root) [(my_workgroup:user)@login01 ~]$
 </code> </code>
  
Line 38: Line 38:
  
 <code bash> <code bash>
-(root) [(it_nss:frey)@login01 keras]$ conda create --prefix=${WORKDIR}/sw/keras/20200204-sklearn keras numpy pip+(root) [(my_workgroup:user)@login01 ~]$ conda create --prefix=${WORKDIR}/sw/keras/20200204-sklearn keras numpy pip
 </code> </code>
  
Line 47: Line 47:
 </code> </code>
  
-Answer "y" to create the virtualenv.  If successful, text will be displayed that mentions source'ing a file to activate the virtualenv:  refrain from doing that, and instead use VALET to manage the Keras virtualenv instances.+Answer "y" to create the virtualenv.  If successful, text will be displayed that mentions source'ing a file to activate the virtualenv:  refrain from doing that, and instead use VALET to manage the Keras virtualenv instances.  Rollback the ''intel-python'' environment changes before proceeding:
  
 +<code bash>
 +(root) [(my_workgroup:user)@login01 ~]$ vpkg_rollback
 +[(my_workgroup:user)@login01 ~]$ 
 +</code>
 +
 +Notice the ''(root)'' has disappeared from the prompt, indicating that the baseline virtualenv has been deactivated.
 ===== VALET Package Definition ===== ===== VALET Package Definition =====
  
Line 57: Line 63:
     prefix: /work/my_workgroup/sw/keras     prefix: /work/my_workgroup/sw/keras
     description: KERAS Python environments     description: KERAS Python environments
 +    flags:
 +        - no-standard-paths
     actions:     actions:
         - action: source         - action: source
           script:           script:
-              sh: anaconda-activate.sh+              sh: intel-python.sh
           order: failure-first           order: failure-first
           success: 0           success: 0
Line 70: Line 78:
 </file> </file>
  
-would be added to ''${WORKDIR}/sw/valet/keras.vpkg_yaml''.  If the file already exists, add your new version at the same level as others:+would be added to ''${WORKDIR}/sw/valet/keras.vpkg_yaml''.  If that file already exists, add your new version at the same level as others:
  
 <file keras.vpkg_yaml> <file keras.vpkg_yaml>
Line 76: Line 84:
     prefix: /work/my_workgroup/sw/keras     prefix: /work/my_workgroup/sw/keras
     description: KERAS Python environments     description: KERAS Python environments
 +    flags:
 +        - no-standard-paths
     actions:     actions:
         - action: source         - action: source
           script:           script:
-              sh: anaconda-activate.sh+              sh: intel-python.sh
           order: failure-first           order: failure-first
           success: 0           success: 0
Line 97: Line 107:
 </file> </file>
  
-<note tip>On Caviness, VALET consults ''${WORKDIR}/sw/valet'' by default after the user has used the ''workgroup'' command.</note>+<note tip>On Caviness after a user has used the ''workgroup'' command, VALET searches for package definitions in ''${WORKDIR}/sw/valet'' by default.</note> 
 + 
 +===== Install SKLearn ===== 
 + 
 +The SKLearn package is not present in the conda online repositories, but it can be installed using ''pip.''  First, activate the new Keras virtualenv: 
 + 
 +<code bash> 
 +[(my_workgroup:user)@login01 ~]$ vpkg_require keras/20200204:sklearn 
 +Adding dependency `intel-python/2019u5:python3` to your environment 
 +Adding package `keras/20200204:sklearn` to your environment 
 +(/work/my_workgroup/sw/keras/20200204-sklearn) [(my_workgroup:user)@login01 keras]$ which pip 
 +/work/my_workgroup/sw/keras/20200204-sklearn/bin/pip 
 +</code> 
 + 
 +A prefix has reappeared on the prompt — the path at which the new Keras virtualenv was created — and the ''pip'' command refers to an executable within that directory tree.  The virtualenv is ready to have SKLearn installed. 
 + 
 +<code bash> 
 +(/work/my_workgroup/sw/keras/20200204-sklearn) [(my_workgroup:user)@login01 ~]$ pip install sklearn 
 +Collecting sklearn 
 +  Downloading .. 
 +    : 
 +Successfully built sklearn 
 +Installing collected packages: joblib, scikit-learn, sklearn 
 +Successfully installed joblib-0.14.1 scikit-learn-0.22.1 sklearn-0.0 
 +(/work/my_workgroup/sw/keras/20200204-sklearn) [(my_workgroup:user)@login01 ~]$ python3 -c "import sklearn;print(sklearn.__version__)" 
 +0.22.1 
 +</code> 
 + 
 +The Keras environment with SKLearn is now ready for use. 
 + 
 +===== Job Scripts ===== 
 + 
 +Any job scripts you submit that want to run scripts using this virtualenv should include something like the following toward its end: 
 + 
 +<code> 
 +# 
 +# Setup Keras virtualenv: 
 +# 
 +vpkg_require keras/20200204:sklearn 
 + 
 +# 
 +# Run a Python script in that virtualenv: 
 +# 
 +python3 my_keras_work.py 
 +rc=$? 
 + 
 +# 
 +# Do cleanup work, etc.... 
 +# 
 + 
 +# 
 +# Exit with whatever exit code our Python script handed back: 
 +# 
 +exit $rc 
 +</code> 
  • technical/recipes/keras-in-virtualenv.1580836920.txt.gz
  • Last modified: 2020-02-04 12:22
  • by frey