Gurobi on Caviness
To determine the available versions of GurobiTM Optimizer installed use
[traine@login00 ~]$ vpkg_versions gurobi Available versions in package (* = default version): [/opt/shared/valet/2.1/etc/gurobi.vpkg_yaml] gurobi The most powerful mathematical optimization solver out there * 9.0.1 release 9.0.1 for Linux
This documentation is based on GurobiTM Optimizer version 9.0.1 and explains how to install and use the Gurobi Python language interface.
Serial or Threaded Python Environment
Adding Gurobi to a Anaconda Environment
Based on the instructions on Gurobi's site for installing the Anaconda Python distribution (https://www.gurobi.com/documentation/9.0/quickstart_linux/ins_the_anaconda_python_di.html#section:Anaconda). The following steps can be adapted from those instructions to create an Anaconda Python environment for running Gurobi.
The workgroup it_css will be used in this example. The anaconda environment will be created in the home directory of user traine. The workgroup ($WORKGROUP) and home directory ($HOME) should be changed to reflect your workgroup and home directory. If Gurobi is going to be used by multiple people in a workgroup, then it might be worth installing the Gurobi anaconda environment in a workgroup directory ($WORKDIR), instead of installing in each person's home directory. Installing it in a workgroup directory will allow everyone within a workgroup to share that Gurobi anaconda environment, and not have to create their own. See workgroup software installs on Caviness for more details.
[traine@login00 ~]$ workgroup -g it_css [(it_css:traine)@login00 ~]$ vpkg_require gurobi Adding package `gurobi/9.0.1` to your environment [(it_css:traine)@login00 ~]$ vpkg_require anaconda/5.2.0:python3 Adding package `anaconda/5.2.0:python3` to your environment [(it_css:traine)@login00 ~]$ conda config --add channels http://conda.anaconda.org/gurobi [(it_css:traine)@login00 ~]$ conda create -p ~/gurobiproject Solving environment: done ==> WARNING: A newer version of conda exists. <== current version: 4.5.11 latest version: 4.8.3 Please update conda by running $ conda update -n base -c defaults conda ## Package Plan ## environment location: /home/1201/gurobiproject Proceed ([y]/n)? y Preparing transaction: done Verifying transaction: done Executing transaction: done # # To activate this environment, use: # > source activate /home/1201/gurobiproject # # To deactivate an active environment, use: # > source deactivate # [(it_css:traine)@login00 ~]$ source activate $HOME/gurobiproject (/home/1201/gurobiproject)[(it_css:traine)@login00 ~]$ conda install gurobi scipy pandas joblib Solving environment: done ==> WARNING: A newer version of conda exists. <== current version: 4.5.11 latest version: 4.8.3 Please update conda by running $ conda update -n base -c defaults conda ## Package Plan ## environment location: /home/2179/gurobiproject added / updated specs: - gurobi - joblib - pandas - scipy The following NEW packages will be INSTALLED: _libgcc_mutex: 0.1-main blas: 1.0-mkl ca-certificates: 2020.1.1-0 certifi: 2019.11.28-py37_1 gurobi: 9.0.1-py37_0 gurobi intel-openmp: 2020.0-166 joblib: 0.14.1-py_0 ld_impl_linux-64: 2.33.1-h53a641e_7 libedit: 3.1.20181209-hc058e9b_0 libffi: 3.2.1-hd88cf55_4 libgcc-ng: 9.1.0-hdf63c60_0 libgfortran-ng: 7.3.0-hdf63c60_0 libstdcxx-ng: 9.1.0-hdf63c60_0 mkl: 2020.0-166 mkl-service: 2.3.0-py37he904b0f_0 mkl_fft: 1.0.15-py37ha843d7b_0 mkl_random: 1.1.0-py37hd6b4f25_0 ncurses: 6.2-he6710b0_0 numpy: 1.18.1-py37h4f9e942_0 numpy-base: 1.18.1-py37hde5b4d6_1 openssl: 1.1.1e-h7b6447c_0 pandas: 1.0.3-py37h0573a6f_0 pip: 20.0.2-py37_1 python: 3.7.7-hcf32534_0_cpython python-dateutil: 2.8.1-py_0 pytz: 2019.3-py_0 readline: 8.0-h7b6447c_0 scipy: 1.4.1-py37h0b6359f_0 setuptools: 46.1.3-py37_0 six: 1.14.0-py37_0 sqlite: 3.31.1-h7b6447c_0 tk: 8.6.8-hbc83047_0 wheel: 0.34.2-py37_0 xz: 5.2.4-h14c3975_4 zlib: 1.2.11-h7b6447c_3 Proceed ([y]/n)? y Preparing transaction: done Verifying transaction: done Executing transaction: done [(it_css:traine)@login00 ~]$ source deactivate
Running a Gurobi Serial or Threaded Job
The below steps are based on copying a template job script and modifying serial.qs or threads.qs script to load run your Gurobi Anaconda environment.
/opt/shared/templates/slurm/ for updated or new templates to use as job scripts to run generic or specific applications designed to provide the best performance on Caviness.
To start, use VALET to load Gurobi and Anaconda. Once those two packages are loaded, the Gurobi Anaconda environment will need to be activated. After the Gurobi Anaconda environment is activated, then the python script that uses Gurobi can be called and ran. After the python script has completed, then the Gurobi Anaconda environment will be need to be deactivated.
Example serial.qs script
# # Do general job environment setup: # . /opt/shared/slurm/templates/libexec/common.sh # # [EDIT] Add your script statements hereafter, or execute a script or program # using the srun command. # vpkg_require gurobi vpkg_require anaconda/5.2.0:python3 source activate ~${USER}/gurobiproject #Directory where the Gurobi Anaconda Environment was installed python3 serial_test.py source deactivate
Example threads.qs script
# # Do standard OpenMP environment setup: # . /opt/shared/slurm/templates/libexec/openmp.sh # # [EDIT] Execute your OpenMP/threaded program using the srun command: # vpkg_require gurobi vpkg_require anaconda/5.2.0:python3 source activate ~${USER}/gurobiproject #Directory where the Gurobi Anaconda Environment was installed python3 sample_parallel.py source deactivate
MPI Python Environment
Adding Gurobi and mpi4py to a Python Environment
Using Gurobi with MPI requires mpi4py and in order for MPI to work properly on our cluster and to understand our networking, it must be compiled using our the version of Open MPI compiled for our cluster.
The instructions below are intended to create a virtual python environment that includes Gurobi and mpi4py in your home directory (or your workgroup) based on the account traine and /home/1201, and workgroup it_css. You will need to adjust the instructions for your own account and workgroup.
Create a directory in which you can organize the Gurobi and mpi4py environment. Make sure you are in your workgroup before you start:
[traine@login01 ~]$ workgroup -g it_css [(it_css:traine)@login01 ~]$ mkdir gurobi-mpi [(it_css:traine)@login01 gurobi-mpi]$ cd gurobi-mpi
Now, we'll use the Intel Python distribution to build a Gurobi 9 environment — without mpi4py at this point! Rather than adding the Gurobi channel to our global conda configuration, we'll enable it just on the new environment itself:
[(it_css:traine)@login01 gurobi-mpi]$ vpkg_require intel-python/2019u5:python3 Adding package `intel-python/2019u5:python3` to your environment (root) [(it_css:anita)@login01 gurobi-mpi]$ conda create --prefix=$(pwd)/envs/20200415 --channel http://conda.anaconda.org/gurobi --channel=intel gurobi=9 scipy pandas joblib python=3 Fetching package metadata ............... Solving package specifications: . Package plan for installation in environment /home/1201/gurobi-mpi/envs/20200415: The following NEW packages will be INSTALLED: bzip2: 1.0.8-1 intel certifi: 2019.11.28-py37_0 intel cycler: 0.10.0-py37_7 intel freetype: 2.10.1-1 intel gurobi: 9.0.1-py37_0 gurobi : scipy-1.3.3-py 100% |########################################################################| Time: 0:00:00 43.73 MB/s pandas-0.25.3- 100% |########################################################################| Time: 0:00:00 44.97 MB/s # # To activate this environment, use: # source activate /home/1201/gurobi-mpi/envs/20200415 # # To deactivate an active environment, use: # source deactivate #
Now activate the new environment:
(root) [(it_css:traine)@login01 gurobi-mpi]$ source activate /home/1201/gurobi-mpi/envs/20200415 (/home/1201/gurobi-mpi/envs/20200415) [(it_css:traine)@login01 gurobi-mpi]$
Note your prompt now shows the virtual python environment you have activated (/home/1201/gurobi-mpi/envs/20200415). Next we need to setup to build mpi4py. The binutils/2.25.1 package is needed because the Anaconda environment's Python was built on a toolset that supported (and used) the –sysroot option to the ld command; the native binutils in CentOS do NOT include that functionality. Naturally, one of our Open MPI package is required to build the binary components of mpi4py:
(/home/1201/gurobi-mpi/envs/20200415) [(it_css:traine)@login01 gurobi-mpi]$ vpkg_require binutils/2.38 openmpi/4.1.2 Adding package `binutils/2.38` to your environment Adding dependency `libfabric/1.13.2` to your environment Adding package `openmpi/4.1.2` to your environment (/home/1201/gurobi-mpi/envs/20200415) [(it_css:traine)@login01 gurobi-mpi]$ pip install --no-binary :all: --compile mpi4py Collecting mpi4py Using cached https://files.pythonhosted.org/packages/ec/8f/bbd8de5ba566dd77e408d8136e2bab7fdf2b97ce06cab830ba8b50a2f588/mpi4py-3.1.3.tar.gz Skipping wheel build for mpi4py, due to binaries being disabled for it. Installing collected packages: mpi4py Running setup.py install for mpi4py ... done Successfully installed mpi4py-3.1.3
Now rollback all the packages in VALET which also deactivates your environment too and then return to your home directory:
(/home/1201/gurobi-mpi/envs/20200415) [(it_css:traine)@login01 gurobi-mpi]$ vpkg_rollback all [(it_css:traine)@login01 gurobi-mpi]$ cd [(it_css:traine)@login01 ~]$
Creating a VALET package definition
Now, create a VALET package to handle the environment setup/teardown and dependencies by using nano or vi to create the file ~/.valet/gurobi-mpi.vpkg_yaml and copy the following text into it changing the /home/1201 to your home directory:
gurobi-mpi:
prefix: /home/1201/gurobi-mpi/envs
description: Gurobi and mpi4py Python environments
actions:
- action: source
script:
sh: anaconda-activate.sh
success: 0
versions:
"20200415":
description: Python 3.7 with scipy, pandas, joblib, and mpi4py
dependencies:
- openmpi/3.1.3
- intel-python/2019u5:python3
- gurobi/9.0.1
nano ~/.valet/gurobi-mpi.vpkg_yaml assumes the directory .valet exists. You will get an error "No such file or directory" when you go to save, if the directory .valet doesn't exist.
Check if the .valet directory exists or not, here it shows the directory exists
[(it_css:traine)@login01 ~]$ file .valet .valet: directory
and in this case the directory does not exist, so create it:
[(it_css:traine)@login01 ~]$ file .valet .valet: cannot open (No such file or directory) [(it_css:traine)@login01 ~]$ mkdir .valet
Thereafter, here is a simple test using VALET to load the environment, test we have the correct version of python3, loading python3 and importing the packages gurobipy and mpi4py, exiting python3, and deactivating the environment:
[(it_css:traine)@login01 ~]$ vpkg_require gurobi-mpi/20200415 Adding dependency `libfabric/1.9.0` to your environment Adding dependency `openmpi/3.1.3` to your environment Adding dependency `intel-python/2019u5:python3` to your environment Adding dependency `gurobi/9.0.1` to your environment Adding package `gurobi-env/20200415` to your environment (/home/1201/gurobi-mpi/envs/20200415) [(it_css:traine)@login01 ~]$ which python3 /home/1201/gurobi-mpi/envs/20200415/bin/python3 (/home/1201/gurobi-mpi/envs/20200415) [(it_css:traine)@login01 ~]$ python3 Python 3.7.7 (default, Mar 26 2020, 15:48:22) [GCC 7.3.0] :: Anaconda, Inc. on linux Type "help", "copyright", "credits" or "license" for more information. >>> import gurobipy >>> import mpi4py >>> (/home/1201/gurobi-mpi/envs/20200415) [(it_css:traine)@login01 ~]$ vpkg_rollback [(it_css:traine)@login01 ~]$
Running a Gurobi MPI Job
First copy the template job script openmpi.qs and modify it to setup the number of cores, memory and time, and use VALET to load your Gurobi and mpi4py environment that you will need to run your job.
/opt/shared/templates/slurm/ for updated or new templates to use as job scripts to run generic or specific applications, designed to provide the best performance on Caviness.
The openmpi.qs file can be found in opt/templates/slurm/generic/mpi directory.