software:gurobi:caviness

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
software:gurobi:caviness [2020-04-16 19:04] – [MPI Python Environment] anitasoftware:gurobi:caviness [2022-07-01 13:04] (current) – frey
Line 1: Line 1:
 ====== Gurobi on Caviness ====== ====== Gurobi on Caviness ======
 +
 +To determine the available versions of Gurobi<sup>TM</sup> Optimizer installed use
  
 <code bash> <code bash>
Line 10: Line 12:
 </code> </code>
  
 +This documentation is based on [[https://www.gurobi.com/documentation/9.0/refman/index.html|Gurobi]]<sup>TM</sup> Optimizer version 9.0.1 and explains how to install and use the Gurobi Python language interface.
 ===== Serial or Threaded Python Environment ===== ===== Serial or Threaded Python Environment =====
  
-==== Adding Gurobi to a Anaconda Environment ===+==== Adding Gurobi to a Anaconda Environment ====
  
 Based on the instructions on Gurobi's site for installing the Anaconda Python distribution (https://www.gurobi.com/documentation/9.0/quickstart_linux/ins_the_anaconda_python_di.html#section:Anaconda). The following steps can be adapted from those instructions to create an Anaconda Python environment for running Gurobi. Based on the instructions on Gurobi's site for installing the Anaconda Python distribution (https://www.gurobi.com/documentation/9.0/quickstart_linux/ins_the_anaconda_python_di.html#section:Anaconda). The following steps can be adapted from those instructions to create an Anaconda Python environment for running Gurobi.
Line 131: Line 134:
 The below steps are based on copying a template job script and modifying ''serial.qs'' or ''threads.qs'' script to load run your Gurobi Anaconda environment.  The below steps are based on copying a template job script and modifying ''serial.qs'' or ''threads.qs'' script to load run your Gurobi Anaconda environment. 
  
-<note tip>It is a good idea to periodically check in ''/opt/templates/slurm/'' for updated or new [[technical:slurm:templates:start|templates]] to use as job scripts to run generic or specific applications, designed to provide the best performance on Caviness.</note>+<note tip>It is a good idea to periodically check in ''/opt/shared/templates/slurm/'' for updated or new [[technical:slurm:caviness:templates:start|templates]] to use as job scripts to run generic or specific applications designed to provide the best performance on Caviness.</note>
  
 To start, use VALET to load Gurobi and Anaconda. Once those two packages are loaded, the Gurobi Anaconda environment will need to be activated. After the Gurobi Anaconda environment is activated, then the python script that uses Gurobi can be called and ran. After the python script has completed, then the Gurobi Anaconda environment will be need to be deactivated. To start, use VALET to load Gurobi and Anaconda. Once those two packages are loaded, the Gurobi Anaconda environment will need to be activated. After the Gurobi Anaconda environment is activated, then the python script that uses Gurobi can be called and ran. After the python script has completed, then the Gurobi Anaconda environment will be need to be deactivated.
Line 152: Line 155:
  
 source activate ~${USER}/gurobiproject #Directory where the Gurobi Anaconda Environment was installed source activate ~${USER}/gurobiproject #Directory where the Gurobi Anaconda Environment was installed
-python serial_test.py+python3 serial_test.py
 source deactivate source deactivate
  
Line 171: Line 174:
  
 source activate ~${USER}/gurobiproject #Directory where the Gurobi Anaconda Environment was installed source activate ~${USER}/gurobiproject #Directory where the Gurobi Anaconda Environment was installed
-python sample_parallel.py+python3 sample_parallel.py
 source deactivate source deactivate
  
Line 177: Line 180:
  
 ===== MPI Python Environment ===== ===== MPI Python Environment =====
 +
 +==== Adding Gurobi and mpi4py to a Python Environment ====
  
 Using Gurobi with MPI requires mpi4py and in order for MPI to work properly on our cluster and to understand our networking, it must be compiled using our the version of Open MPI compiled for our cluster. Using Gurobi with MPI requires mpi4py and in order for MPI to work properly on our cluster and to understand our networking, it must be compiled using our the version of Open MPI compiled for our cluster.
Line 196: Line 201:
  Adding package `intel-python/2019u5:python3` to your environment  Adding package `intel-python/2019u5:python3` to your environment
  
- (root) [(it_css:anita)@login01  gurobi-mpi]$ conda create --prefix=$(pwd)/envs/20200415 --channel http://conda.anaconda.org/gurobi gurobi=9 scipy pandas joblib python=3+ (root) [(it_css:anita)@login01  gurobi-mpi]$ conda create --prefix=$(pwd)/envs/20200415 --channel http://conda.anaconda.org/gurobi --channel=intel gurobi=9 scipy pandas joblib python=3
  Fetching package metadata ...............  Fetching package metadata ...............
  Solving package specifications: .  Solving package specifications: .
Line 234: Line 239:
  
 <code bash> <code bash>
- (/home/1201/gurobi-mpi/envs/20200415)  [(it_css:traine)@login01  gurobi-mpi]$ vpkg_require binutils/2.25.1 openmpi/3.1.3 + (/home/1201/gurobi-mpi/envs/20200415)  [(it_css:traine)@login01  gurobi-mpi]$ vpkg_require binutils/2.38 openmpi/4.1.2 
- Adding package `binutils/2.25.1` to your environment + Adding package `binutils/2.38` to your environment 
- Adding dependency `libfabric/1.9.0` to your environment + Adding dependency `libfabric/1.13.2` to your environment 
- Adding package `openmpi/3.1.3` to your environment+ Adding package `openmpi/4.1.2` to your environment
  
  (/home/1201/gurobi-mpi/envs/20200415)  [(it_css:traine)@login01  gurobi-mpi]$ pip install --no-binary :all: --compile mpi4py  (/home/1201/gurobi-mpi/envs/20200415)  [(it_css:traine)@login01  gurobi-mpi]$ pip install --no-binary :all: --compile mpi4py
  Collecting mpi4py  Collecting mpi4py
-   Using cached https://files.pythonhosted.org/packages/ec/8f/bbd8de5ba566dd77e408d8136e2bab7fdf2b97ce06cab830ba8b50a2f588/mpi4py-3.0.3.tar.gz+   Using cached https://files.pythonhosted.org/packages/ec/8f/bbd8de5ba566dd77e408d8136e2bab7fdf2b97ce06cab830ba8b50a2f588/mpi4py-3.1.3.tar.gz
  Skipping wheel build for mpi4py, due to binaries being disabled for it.  Skipping wheel build for mpi4py, due to binaries being disabled for it.
  Installing collected packages: mpi4py  Installing collected packages: mpi4py
      Running setup.py install for mpi4py ... done      Running setup.py install for mpi4py ... done
- Successfully installed mpi4py-3.0.3+ Successfully installed mpi4py-3.1.3
 </code> </code>
  
Line 256: Line 261:
 </code> </code>
  
-Now, create a VALET package to handle the environment setup/teardown and dependencies by using "nano" or "vi" to create the file ''~/.valet/gurobi-env.vpkg_yaml'' and copy the following text into it changing the ''/home/1201'' to your home directory:+==== Creating a VALET package definition ==== 
 +Now, create a VALET package to handle the environment setup/teardown and dependencies by using ''nano'' or ''vi'' to create the file ''~/.valet/gurobi-mpi.vpkg_yaml'' and copy the following text into it changing the ''/home/1201'' to your home directory:
  
 <code bash> <code bash>
Line 278: Line 284:
 </code> </code>
  
-<note important>If you use ''nano ~/.valet/gurobi-env.vpkg_yaml'', it assumes you already have the directory ''.valet''. You will get an error no such file or directory when you go to save, if the directory ''.valet'' doesn't exist. Check it first and create it before trying to use nano. +<note important>Using ''nano ~/.valet/gurobi-mpi.vpkg_yaml'' assumes the directory ''.valet'' exists. You will get an error "No such file or directory" when you go to save, if the directory ''.valet'' doesn't exist. 
  
-If the ''.valet'' directory exists:+Check if the ''.valet'' directory exists or not, here it shows the directory exists
  
 <code bash> <code bash>
Line 287: Line 293:
 </code> </code>
  
-If the ''.valet'' directory does not exist, so create it:+and in this case the directory does not exist, so create it:
  
 <code bash> <code bash>
Line 296: Line 302:
 </note> </note>
  
-Thereafter, here is a simple test using VALET to load the environment and test we have the correct version of python3:+Thereafter, here is a simple test using VALET to load the environment, test we have the correct version of python3, loading python3 and importing the packages gurobipy and mpi4py, exiting python3, and deactivating the environment:
  
 <code bash> <code bash>
-[(it_css:traine)@login01  ~]$ vpkg_require gurobi-env/20200415+[(it_css:traine)@login01  ~]$ vpkg_require gurobi-mpi/20200415
 Adding dependency `libfabric/1.9.0` to your environment Adding dependency `libfabric/1.9.0` to your environment
 Adding dependency `openmpi/3.1.3` to your environment Adding dependency `openmpi/3.1.3` to your environment
Line 305: Line 311:
 Adding dependency `gurobi/9.0.1` to your environment Adding dependency `gurobi/9.0.1` to your environment
 Adding package `gurobi-env/20200415` to your environment Adding package `gurobi-env/20200415` to your environment
- 
 (/home/1201/gurobi-mpi/envs/20200415) [(it_css:traine)@login01 ~]$ which python3 (/home/1201/gurobi-mpi/envs/20200415) [(it_css:traine)@login01 ~]$ which python3
 /home/1201/gurobi-mpi/envs/20200415/bin/python3 /home/1201/gurobi-mpi/envs/20200415/bin/python3
 +(/home/1201/gurobi-mpi/envs/20200415) [(it_css:traine)@login01 ~]$ python3
 +Python 3.7.7 (default, Mar 26 2020, 15:48:22)
 +[GCC 7.3.0] :: Anaconda, Inc. on linux
 +Type "help", "copyright", "credits" or "license" for more information.
 +>>> import gurobipy
 +>>> import mpi4py
 +>>>
 +(/home/1201/gurobi-mpi/envs/20200415) [(it_css:traine)@login01 ~]$ vpkg_rollback
 +[(it_css:traine)@login01 ~]$
 </code> </code>
  
Line 314: Line 328:
 First copy the template job script ''openmpi.qs'' and modify it to setup the number of cores, memory and time, and use VALET to load your Gurobi and mpi4py environment that you will need to run your job.  First copy the template job script ''openmpi.qs'' and modify it to setup the number of cores, memory and time, and use VALET to load your Gurobi and mpi4py environment that you will need to run your job. 
  
-<note tip>It is a good idea to periodically check in ''/opt/templates/slurm/'' for updated or new [[technical:slurm:templates:start|templates]] to use as job scripts to run generic or specific applications, designed to provide the best performance on Caviness.</note>+<note tip>It is a good idea to periodically check in ''/opt/shared/templates/slurm/'' for updated or new [[technical:slurm:templates:start|templates]] to use as job scripts to run generic or specific applications, designed to provide the best performance on Caviness.</note>
  
 The ''openmpi.qs'' file can be found in ''opt/templates/slurm/generic/mpi'' directory. The ''openmpi.qs'' file can be found in ''opt/templates/slurm/generic/mpi'' directory.
  
  • software/gurobi/caviness.1587078240.txt.gz
  • Last modified: 2020-04-16 19:04
  • by anita