technical:recipes:mpi4py-in-virtualenv

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
Next revisionBoth sides next revision
technical:recipes:mpi4py-in-virtualenv [2024-03-05 17:02] – [MPI and Conda Variants] anitatechnical:recipes:mpi4py-in-virtualenv [2024-03-07 15:04] – [MPI and Conda Variants] anita
Line 15: Line 15:
  
 <WRAP center round info 60%> <WRAP center round info 60%>
-On Caviness and DARWIN we would likely choose the Intel Python distribution selection a Python 3 version over Anaconda (e.g. ''vpkg_versions intel-python'') since it automatically enables Intel's distribution channel.  That channel includes Numpy built against the Intel MKL library, for example, and other highly-optimized variants of computationally-intensive Python components.+On Caviness and DARWIN we would likely choose the Intel distribution selecting a Python 3 version over Anaconda (e.g. ''vpkg_versions intel-oneapi'' or ''intel-python'') since it automatically enables Intel's distribution channel.  That channel includes Numpy built against the Intel MKL library, for example, and other highly-optimized variants of computationally-intensive Python components.
  
 As of November 2020, the majority of packages populating the Intel channel require baseline operating system libraries (like ''glibc'') newer than what Farber provides:  a clear example of the binary compatibility issues that are present in conda software distribution. As of November 2020, the majority of packages populating the Intel channel require baseline operating system libraries (like ''glibc'') newer than what Farber provides:  a clear example of the binary compatibility issues that are present in conda software distribution.
Line 292: Line 292:
 $ which mpirun $ which mpirun
 /opt/shared/openmpi/4.0.2/bin/mpirun /opt/shared/openmpi/4.0.2/bin/mpirun
 +</code>
 +
 +===== DARWIN =====
 +
 +The steps for completing this work on DARWIN are similar to those presented for Caviness and of course following the first part of the directory structure setup presented for Farber.  We will instead use the Intel oneAPI Python distribution:
 +
 +<code bash>
 +$ vpkg_require intel-oneapi/2024
 +Adding dependency `gcc/12.2.0` to your environment
 +Adding package `intel-oneapi/2024.0.1.46` to your environment
 +</code>
 +
 +==== Create the Virtual Environment ====
 +
 +The virtual environment is first populated with all packages that **do not** require mpi4py.  Any packages requiring mpi4py must be installed //after// we build and install our local copy of mpi4py in the virtual environment.  In this example, neither Numpy nor Scipy require mpi4py.
 +
 +<code bash>
 +$ conda create --prefix ${HOME}/conda-envs/my-sci-app/20240307 --channel intel --override-channels python'=>3.9' numpy scipy
 +Collecting package metadata (current_repodata.json): done
 +Solving environment: done
 +    :
 +Proceed ([y]/n)? y
 +    :
 +#
 +# To activate this environment, use
 +#
 +#     $ conda activate /home/1006/conda-envs/my-sci-app/20240307
 +#
 +# To deactivate an active environment, use
 +#
 +#     $ conda deactivate
 +
 +</code>
 +
 +Before building and installing mpi4py the environment needs to be activated:
 +
 +<code bash>
 +$ conda activate /home/1006/conda-envs/my-sci-app/20240307
 +(/home/1006/conda-envs/my-sci-app/20240307)$
 +</code>
 +
 +==== Building mpi4py ====
 +
 +With the new virtual environment activated, we can now build mpi4py against the local Open MPI library we added to the shell environment.
 +
 +<code base>
 +(/home/1006/conda-envs/my-sci-app/20240307)$ pip install --no-binary :all: --compile mpi4py
 +Collecting mpi4py
 +  Downloading mpi4py-3.1.5.tar.gz (2.5 MB)
 +     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 16.7 MB/s eta 0:00:00
 +  Installing build dependencies ... done
 +  Getting requirements to build wheel ... done
 +  Preparing metadata (pyproject.toml) ... done
 +Building wheels for collected packages: mpi4py
 +  Building wheel for mpi4py (pyproject.toml) ... done
 +  Created wheel for mpi4py: filename=mpi4py-3.1.5-cp310-cp310-linux_x86_64.whl size=634821 sha256=78a58c10acd22b3cf2ebf9e73b445d6775ac29f3f59c37e63bd16e27b7467ba2
 +  Stored in directory: /home/1006/.cache/pip/wheels/18/2b/7f/c852523089e9182b45fca50ff56f49a51eeb6284fd25a66713
 +Successfully built mpi4py
 +Installing collected packages: mpi4py
 +Successfully installed mpi4py-3.1.5
 +</code>
 +
 +The ''--no-binary :all:'' flag prohibits the installation of any packages that include binary components, effectively forcing a rebuild of mpi4py from source.  The ''--compile'' flag pre-processes all Python scripts in the mpi4py package (versus allowing them to be processed and cached later).  The environment now includes support for mpi4py linked against the ''Intel oneAPI mpi'' library on DARWIN:
 +
 +<code bash>
 +(/home/1006/conda-envs/my-sci-app/20240307)$ pip list | grep mpi4py
 +mpi4py             3.1.5
 +</code>
 +
 +Additional packages that require mpi4py can now be installed into the environment.
 +
 +==== VALET Package Definition ====
 +
 +The new virtual environment can easily be added to your login shell and job runtime environments using VALET.  First, ensure you have your personal VALET package definition directory present:
 +
 +<code bash>
 +$ mkdir -p ${HOME}/.valet
 +$ echo ${HOME}/conda-envs/my-sci-app
 +/home/1006/conda-envs/my-sci-app
 +</code>
 +
 +Take note of the path echoed, then create a new file named ''${HOME}/.valet/my-sci-app.vpkg_yaml'' and add the following text to it:
 +
 +<code yaml>
 +my-sci-app:
 +    prefix: /home/1006/conda-envs/my-sci-app
 +    description: Some scientific app project in Python
 +    flags:
 +        - no-standard-paths
 +    actions:
 +        - action: source
 +          script:
 +              sh: anaconda-activate.sh
 +          order: failure-first
 +          success: 0
 +    versions:
 +          "20240307":
 +              description: environment built Mar 7, 2024
 +              dependencies:
 +                  - intel-oneapi/2024
 +</code>
 +
 +=== Using the Virtual Environment ===
 +
 +The versions of the virtual environment declared in the VALET package are listed using the ''vpkg_versions'' command:
 +
 +<code bash>
 +$ vpkg_versions my-sci-app
 +
 +Available versions in package (* = default version):
 +
 +[/home/1006/.valet/my-sci-app.vpkg_yaml]
 +my-sci-app  Some scientific app project in Python
 +* 20240307  environment built Mar 7, 2024
 +</code>
 +
 +Activating the virtual environment is accomplished using the ''vpkg_require'' command (in your login shell or inside job scripts):
 +
 +<code bash>
 +$ vpkg_require my-sci-app/20240307
 +Adding dependency `gcc/12.2.0` to your environment
 +Adding dependency `intel-oneapi/2024.0.1.46` to your environment
 +Adding package `my-sci-app/20240307` to your environment
 +(/home/1006/conda-envs/my-sci-app/20240307)$ which python3
 +~/conda-envs/my-sci-app/20240305/bin/python3
 +(/home/1006/conda-envs/my-sci-app/20240305)$ pip list | grep mpi4py
 +mpi4py             3.1.5
 +(/home/1006/conda-envs/my-sci-app/20240305)$ which mpirun
 +/opt/shared/intel-oneapi/2024.0.1.46/mpi/2021.11/bin/mpirun
 </code> </code>
  
  • technical/recipes/mpi4py-in-virtualenv.txt
  • Last modified: 2024-03-07 17:04
  • by anita