Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revisionLast revisionBoth sides next revision | ||
software:gurobi:caviness [2020-04-16 19:04] – [MPI Python Environment] anita | software:gurobi:caviness [2022-05-05 11:49] – anita | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== Gurobi on Caviness ====== | ||
- | |||
- | <code bash> | ||
- | [traine@login00 ~]$ vpkg_versions gurobi | ||
- | Available versions in package (* = default version): | ||
- | |||
- | [/ | ||
- | gurobi | ||
- | * 9.0.1 release 9.0.1 for Linux | ||
- | </ | ||
- | |||
- | ===== Serial or Threaded Python Environment ===== | ||
- | |||
- | ==== Adding Gurobi to a Anaconda Environment === | ||
- | |||
- | Based on the instructions on Gurobi' | ||
- | |||
- | The workgroup '' | ||
- | |||
- | <code bash> | ||
- | [traine@login00 ~]$ workgroup -g it_css | ||
- | [(it_css: | ||
- | Adding package `gurobi/ | ||
- | [(it_css: | ||
- | Adding package `anaconda/ | ||
- | [(it_css: | ||
- | [(it_css: | ||
- | Solving environment: | ||
- | |||
- | |||
- | ==> WARNING: A newer version of conda exists. <== | ||
- | current version: 4.5.11 | ||
- | latest version: 4.8.3 | ||
- | |||
- | Please update conda by running | ||
- | |||
- | $ conda update -n base -c defaults conda | ||
- | |||
- | |||
- | |||
- | ## Package Plan ## | ||
- | |||
- | environment location: / | ||
- | |||
- | |||
- | Proceed ([y]/n)? y | ||
- | Preparing transaction: | ||
- | Verifying transaction: | ||
- | Executing transaction: | ||
- | # | ||
- | # To activate this environment, | ||
- | # > source activate / | ||
- | # | ||
- | # To deactivate an active environment, | ||
- | # > source deactivate | ||
- | # | ||
- | |||
- | [(it_css: | ||
- | (/ | ||
- | Solving environment: | ||
- | |||
- | |||
- | ==> WARNING: A newer version of conda exists. <== | ||
- | current version: 4.5.11 | ||
- | latest version: 4.8.3 | ||
- | |||
- | Please update conda by running | ||
- | |||
- | $ conda update -n base -c defaults conda | ||
- | |||
- | |||
- | |||
- | ## Package Plan ## | ||
- | |||
- | environment location: / | ||
- | |||
- | added / updated specs: | ||
- | - gurobi | ||
- | - joblib | ||
- | - pandas | ||
- | - scipy | ||
- | |||
- | |||
- | The following NEW packages will be INSTALLED: | ||
- | |||
- | _libgcc_mutex: | ||
- | blas: | ||
- | ca-certificates: | ||
- | certifi: | ||
- | gurobi: | ||
- | intel-openmp: | ||
- | joblib: | ||
- | ld_impl_linux-64: | ||
- | libedit: | ||
- | libffi: | ||
- | libgcc-ng: | ||
- | libgfortran-ng: | ||
- | libstdcxx-ng: | ||
- | mkl: 2020.0-166 | ||
- | mkl-service: | ||
- | mkl_fft: | ||
- | mkl_random: | ||
- | ncurses: | ||
- | numpy: | ||
- | numpy-base: | ||
- | openssl: | ||
- | pandas: | ||
- | pip: 20.0.2-py37_1 | ||
- | python: | ||
- | python-dateutil: | ||
- | pytz: | ||
- | readline: | ||
- | scipy: | ||
- | setuptools: | ||
- | six: 1.14.0-py37_0 | ||
- | sqlite: | ||
- | tk: | ||
- | wheel: | ||
- | xz: | ||
- | zlib: | ||
- | |||
- | Proceed ([y]/n)? y | ||
- | |||
- | Preparing transaction: | ||
- | Verifying transaction: | ||
- | Executing transaction: | ||
- | [(it_css: | ||
- | </ | ||
- | ==== Running a Gurobi Serial or Threaded Job ==== | ||
- | |||
- | The below steps are based on copying a template job script and modifying '' | ||
- | |||
- | <note tip>It is a good idea to periodically check in ''/ | ||
- | |||
- | To start, use VALET to load Gurobi and Anaconda. Once those two packages are loaded, the Gurobi Anaconda environment will need to be activated. After the Gurobi Anaconda environment is activated, then the python script that uses Gurobi can be called and ran. After the python script has completed, then the Gurobi Anaconda environment will be need to be deactivated. | ||
- | |||
- | |||
- | === Example serial.qs script === | ||
- | <code bash> | ||
- | # | ||
- | # Do general job environment setup: | ||
- | # | ||
- | . / | ||
- | |||
- | # | ||
- | # [EDIT] Add your script statements hereafter, or execute a script or program | ||
- | # using the srun command. | ||
- | # | ||
- | |||
- | vpkg_require gurobi | ||
- | vpkg_require anaconda/ | ||
- | |||
- | source activate ~${USER}/ | ||
- | python serial_test.py | ||
- | source deactivate | ||
- | |||
- | </ | ||
- | === Example threads.qs script === | ||
- | <code bash> | ||
- | # | ||
- | # Do standard OpenMP environment setup: | ||
- | # | ||
- | . / | ||
- | |||
- | # | ||
- | # [EDIT] Execute your OpenMP/ | ||
- | # | ||
- | |||
- | vpkg_require gurobi | ||
- | vpkg_require anaconda/ | ||
- | |||
- | source activate ~${USER}/ | ||
- | python sample_parallel.py | ||
- | source deactivate | ||
- | |||
- | </ | ||
- | |||
- | ===== MPI Python Environment ===== | ||
- | |||
- | Using Gurobi with MPI requires mpi4py and in order for MPI to work properly on our cluster and to understand our networking, it must be compiled using our the version of Open MPI compiled for our cluster. | ||
- | |||
- | The instructions below are intended to create a virtual python environment that includes Gurobi and mpi4py in your home directory (or your workgroup) based on the account '' | ||
- | |||
- | Create a directory in which you can organize the Gurobi and mpi4py environment. Make sure you are in your workgroup before you start: | ||
- | |||
- | <code bash> | ||
- | [traine@login01 ~]$ workgroup -g it_css | ||
- | [(it_css: | ||
- | [(it_css: | ||
- | </ | ||
- | |||
- | Now, we'll use the Intel Python distribution to build a Gurobi 9 environment — without mpi4py at this point! | ||
- | |||
- | <code bash> | ||
- | | ||
- | | ||
- | |||
- | | ||
- | | ||
- | | ||
- | |||
- | | ||
- | |||
- | The following NEW packages will be INSTALLED: | ||
- | |||
- | | ||
- | | ||
- | | ||
- | | ||
- | |||
- | | ||
- | |||
- | : | ||
- | |||
- | | ||
- | | ||
- | # | ||
- | # To activate this environment, | ||
- | # | ||
- | # | ||
- | # To deactivate an active environment, | ||
- | # | ||
- | # | ||
- | </ | ||
- | |||
- | Now activate the new environment: | ||
- | |||
- | <code bash> | ||
- | (root) [(it_css: | ||
- | (/ | ||
- | </ | ||
- | |||
- | Note your prompt now shows the virtual python environment you have activated '' | ||
- | |||
- | <code bash> | ||
- | | ||
- | | ||
- | | ||
- | | ||
- | |||
- | | ||
- | | ||
- | Using cached https:// | ||
- | | ||
- | | ||
- | | ||
- | | ||
- | </ | ||
- | |||
- | Now rollback all the packages in VALET which also deactivates your environment too and then return to your home directory: | ||
- | |||
- | <code bash> | ||
- | | ||
- | | ||
- | | ||
- | </ | ||
- | |||
- | Now, create a VALET package to handle the environment setup/ | ||
- | |||
- | <code bash> | ||
- | | ||
- | | ||
- | | ||
- | |||
- | | ||
- | - action: source | ||
- | | ||
- | sh: anaconda-activate.sh | ||
- | | ||
- | |||
- | | ||
- | " | ||
- | | ||
- | | ||
- | - openmpi/ | ||
- | - intel-python/ | ||
- | - gurobi/ | ||
- | </ | ||
- | |||
- | <note important> | ||
- | |||
- | If the '' | ||
- | |||
- | <code bash> | ||
- | [(it_css: | ||
- | .valet: directory | ||
- | </ | ||
- | |||
- | If the '' | ||
- | |||
- | <code bash> | ||
- | [(it_css: | ||
- | .valet: cannot open (No such file or directory) | ||
- | [(it_css: | ||
- | </ | ||
- | </ | ||
- | |||
- | Thereafter, here is a simple test using VALET to load the environment and test we have the correct version of python3: | ||
- | |||
- | <code bash> | ||
- | [(it_css: | ||
- | Adding dependency `libfabric/ | ||
- | Adding dependency `openmpi/ | ||
- | Adding dependency `intel-python/ | ||
- | Adding dependency `gurobi/ | ||
- | Adding package `gurobi-env/ | ||
- | |||
- | (/ | ||
- | / | ||
- | </ | ||
- | |||
- | ==== Running a Gurobi MPI Job ==== | ||
- | |||
- | First copy the template job script '' | ||
- | |||
- | <note tip>It is a good idea to periodically check in ''/ | ||
- | |||
- | The '' | ||