technical:recipes:mpi4py-in-virtualenv

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
technical:recipes:mpi4py-in-virtualenv [2024-03-07 16:51] – [DARWIN] anitatechnical:recipes:mpi4py-in-virtualenv [2025-01-31 15:29] (current) – [VALET Package Definition] anita
Line 15: Line 15:
  
 <WRAP center round info 60%> <WRAP center round info 60%>
-On Caviness and DARWIN we would likely choose the Intel distribution selecting a Python 3 version over Anaconda (e.g. ''vpkg_versions intel-oneapi'' or ''intel-python'') since it automatically enables Intel's distribution channel.  That channel includes Numpy built against the Intel MKL library, for example, and other highly-optimized variants of computationally-intensive Python components.+Due to recent announcements regarding Anaconda, and Intel dropping their distribution channel, any documentation referring to Intel's channel will need to be updated.
  
-As of November 2020, the majority of packages populating the Intel channel require baseline operating system libraries (like ''glibc'') newer than what Farber provides:  a clear example of the binary compatibility issues that are present in conda software distribution.+Please use ''conda-forge'' channel for installations.
 </WRAP> </WRAP>
  
Line 174: Line 174:
  
 <code bash> <code bash>
-$ vpkg_require openmpi/4.0.2 intel-python/2020u2:python3 +$ vpkg_require openmpi/4.1.4:gcc-12.1.0 anaconda/2024.02 
-Adding dependency `libfabric/1.9.0` to your environment +Adding dependency `libfabric/1.13.2` to your environment 
-Adding package `openmpi/4.0.2` to your environment +Adding dependency `binutils/2.35` to your environment 
-Adding package `intel-python/2020u2:python3` to your environment+Adding dependency `gcc/12.1.0` to your environment 
 +Adding package `openmpi/4.1.4:gcc-12.1.0` to your environment 
 +Adding package `anaconda/2024.02` to your environment
 </code> </code>
  
Line 185: Line 187:
  
 <code bash> <code bash>
-$ conda create --prefix ${HOME}/conda-envs/my-sci-app/20201102 --channel intel --override-channels python'=>3.7' numpy scipy+$ conda create --prefix ${HOME}/conda-envs/my-sci-app/20201102 --channel defaults --override-channels python'=>3.7' numpy scipy
 Collecting package metadata (current_repodata.json): done Collecting package metadata (current_repodata.json): done
 Solving environment: done Solving environment: done
Line 211: Line 213:
 ==== Building mpi4py ==== ==== Building mpi4py ====
  
-With the new virtual environment activated, we can now build mpi4py against the local Open MPI library we added to the shell environment.+With the new virtual environment activated, we can now build mpi4py against the local Open MPI library we added to the shell environment. Due to Anaconda trying to use a version of ''ld'' as part of the virtual environment in lieu of the system ''ld'', you need to change the permissions to allow the compile to work properly.
  
 <code base> <code base>
 +(/home/1001/conda-envs/my-sci-app/20201102)$ chmod 000 /home/1001/conda-envs/my-sci-app/20201102/compiler_compat/ld
 (/home/1001/conda-envs/my-sci-app/20201102)$ pip install --no-binary :all: --compile mpi4py (/home/1001/conda-envs/my-sci-app/20201102)$ pip install --no-binary :all: --compile mpi4py
 Collecting mpi4py Collecting mpi4py
-  Using cached mpi4py-3.0.3.tar.gz (1.4 MB)+  Using cached mpi4py-4.0.1.tar.gz (466 kB)
 Skipping wheel build for mpi4py, due to binaries being disabled for it. Skipping wheel build for mpi4py, due to binaries being disabled for it.
 Installing collected packages: mpi4py Installing collected packages: mpi4py
     Running setup.py install for mpi4py ... done     Running setup.py install for mpi4py ... done
-Successfully installed mpi4py-3.0.3+Successfully installed mpi4py-4.0.1
 </code> </code>
  
-The ''--no-binary :all:'' flag prohibits the installation of any packages that include binary components, effectively forcing a rebuild of mpi4py from source.  The ''--compile'' flag pre-processes all Python scripts in the mpi4py package (versus allowing them to be processed and cached later).  The environment now includes support for mpi4py linked against the ''openmpi/4.0.2'' library on Caviness:+The ''--no-binary :all:'' flag prohibits the installation of any packages that include binary components, effectively forcing a rebuild of mpi4py from source.  The ''--compile'' flag pre-processes all Python scripts in the mpi4py package (versus allowing them to be processed and cached later).  The environment now includes support for mpi4py linked against the ''openmpi/4.1.4:gcc-12.1.0'' library on Caviness:
  
 <code bash> <code bash>
 (/home/1001/conda-envs/my-sci-app/20201102)$ pip list | grep mpi4py (/home/1001/conda-envs/my-sci-app/20201102)$ pip list | grep mpi4py
-mpi4py      3.0.3+mpi4py      4.0.1
 </code> </code>
  
Line 260: Line 263:
               description: environment built Nov 2, 2020               description: environment built Nov 2, 2020
               dependencies:               dependencies:
-                  - openmpi/4.0.2 +                  - openmpi/4.1.4:gcc-12.1.0 
-                  - intel-python/2020u2:python3+                  - anaconda/2024.02
 </code> </code>
  
Line 282: Line 285:
 <code bash> <code bash>
 $ vpkg_require my-sci-app/20201102 $ vpkg_require my-sci-app/20201102
-Adding dependency `libfabric/1.9.0` to your environment +Adding dependency `libfabric/1.13.2` to your environment 
-Adding dependency `openmpi/4.0.2` to your environment +Adding dependency `binutils/2.35` to your environment 
-Adding dependency `intel-python/2020u2:python3` to your environment+Adding dependency `gcc/12.1.0` to your environment 
 +Adding package `openmpi/4.1.4:gcc-12.1.0` to your environment 
 +Adding package `anaconda/2024.02` to your environment
 Adding package `my-sci-app/20201102` to your environment Adding package `my-sci-app/20201102` to your environment
 (/home/1001/conda-envs/my-sci-app/20201102)$ which python3 (/home/1001/conda-envs/my-sci-app/20201102)$ which python3
 ~/conda-envs/my-sci-app/20201102/bin/python3 ~/conda-envs/my-sci-app/20201102/bin/python3
 (/home/1001/conda-envs/my-sci-app/20201102)$ pip list | grep mpi4py (/home/1001/conda-envs/my-sci-app/20201102)$ pip list | grep mpi4py
-mpi4py      3.0.3+mpi4py      4.0.1
 $ which mpirun $ which mpirun
-/opt/shared/openmpi/4.0.2/bin/mpirun+/opt/shared/openmpi/4.1.4:gcc-12.1.0/bin/mpirun
 </code> </code>
  
Line 299: Line 304:
  
 <code bash> <code bash>
-$ vpkg_require openmpi/5.0.2:intel-oneapi-2024 intel-oneapi/2024+$ vpkg_require openmpi/4.1.5:gcc-12.2 anaconda/2024.02
 Adding dependency `gcc/12.2.0` to your environment Adding dependency `gcc/12.2.0` to your environment
-Adding dependency `intel-oneapi/2024.0.1.46` to your environment 
 Adding dependency `ucx/1.13.1` to your environment Adding dependency `ucx/1.13.1` to your environment
-Adding package `openmpi/5.0.2:intel-oneapi-2024` to your environment+Adding package `openmpi/4.1.5:gcc-12.2` to your environment 
 +Adding package `anaconda/2024.02:python3` to your environment
 </code> </code>
  
Line 311: Line 316:
  
 <code bash> <code bash>
-$ conda create --prefix ${HOME}/conda-envs/my-sci-app/20240307 --channel intel --override-channels python'=>3.9' numpy scipy+$ conda create --prefix ${HOME}/conda-envs/my-sci-app/20250121 --channel defaults --override-channels python'=>3.7' numpy scipy
 Collecting package metadata (current_repodata.json): done Collecting package metadata (current_repodata.json): done
 Solving environment: done Solving environment: done
Line 320: Line 325:
 # To activate this environment, use # To activate this environment, use
 # #
-#     $ conda activate /home/1006/conda-envs/my-sci-app/20240307+#     $ conda activate /home/1006/conda-envs/my-sci-app/20250121
 # #
 # To deactivate an active environment, use # To deactivate an active environment, use
Line 331: Line 336:
  
 <code bash> <code bash>
-$ conda activate /home/1006/conda-envs/my-sci-app/20240307 +$ conda activate /home/1006/conda-envs/my-sci-app/20250121 
-(/home/1006/conda-envs/my-sci-app/20240307)$+(/home/1006/conda-envs/my-sci-app/20250121)$
 </code> </code>
  
Line 340: Line 345:
  
 <code base> <code base>
-(/home/1006/conda-envs/my-sci-app/20240307)$ pip install --no-binary :all: --compile mpi4py+(/home/1006/conda-envs/my-sci-app/20250121)$ chmod 000 /home/1001/conda-envs/my-sci-app/20201102/compiler_compat/ld 
 +(/home/1006/conda-envs/my-sci-app/20250121)$ pip install --no-binary :all: --compile mpi4py 
 +$ pip install --no-binary :all: --compile mpi4py
 Collecting mpi4py Collecting mpi4py
-  Downloading mpi4py-3.1.5.tar.gz (2.5 MB) +  Downloading mpi4py-4.0.1.tar.gz (466 kB)
-     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 16.7 MB/s eta 0:00:00+
   Installing build dependencies ... done   Installing build dependencies ... done
   Getting requirements to build wheel ... done   Getting requirements to build wheel ... done
 +  Installing backend dependencies ... done
   Preparing metadata (pyproject.toml) ... done   Preparing metadata (pyproject.toml) ... done
 Building wheels for collected packages: mpi4py Building wheels for collected packages: mpi4py
   Building wheel for mpi4py (pyproject.toml) ... done   Building wheel for mpi4py (pyproject.toml) ... done
-  Created wheel for mpi4py: filename=mpi4py-3.1.5-cp310-cp310-linux_x86_64.whl size=634821 sha256=78a58c10acd22b3cf2ebf9e73b445d6775ac29f3f59c37e63bd16e27b7467ba2 +  Created wheel for mpi4py: filename=mpi4py-4.0.1-cp313-cp313-linux_x86_64.whl size=997834 sha256=b09b4fe26c8aa940bdcbdb512960fb73edb9ed9ed698b9455db3e1f3d5b078a5 
-  Stored in directory: /home/1006/.cache/pip/wheels/18/2b/7f/c852523089e9182b45fca50ff56f49a51eeb6284fd25a66713+  Stored in directory: /home/1006/.cache/pip/wheels/27/79/62/f500b54e8b8ce5f5e54e7b84e8695938988ca274117d39983b
 Successfully built mpi4py Successfully built mpi4py
 Installing collected packages: mpi4py Installing collected packages: mpi4py
-Successfully installed mpi4py-3.1.5+Successfully installed mpi4py-4.0.1
 </code> </code>
  
-The ''--no-binary :all:'' flag prohibits the installation of any packages that include binary components, effectively forcing a rebuild of mpi4py from source.  The ''--compile'' flag pre-processes all Python scripts in the mpi4py package (versus allowing them to be processed and cached later).  The environment now includes support for mpi4py linked against the ''Intel oneAPI mpi'' library on DARWIN:+The ''--no-binary :all:'' flag prohibits the installation of any packages that include binary components, effectively forcing a rebuild of mpi4py from source.  The ''--compile'' flag pre-processes all Python scripts in the mpi4py package (versus allowing them to be processed and cached later).  The environment now includes support for mpi4py linked against the ''''openmpi/4.1.4:gcc-12.2.0'''' library on DARWIN:
  
 <code bash> <code bash>
-(/home/1006/conda-envs/my-sci-app/20240307)$ pip list | grep mpi4py +(/home/1006/conda-envs/my-sci-app/20250121)$ pip list | grep mpi4py 
-mpi4py             3.1.5+mpi4py      4.0.1
 </code> </code>
  
Line 390: Line 397:
           success: 0           success: 0
     versions:     versions:
-          "20240307": +          "20250121": 
-              description: environment built Mar 7, 2024+              description: environment built Jan 21, 2025
               dependencies:               dependencies:
-                  - intel-oneapi/2024+                  - openmpi/4.1.5:gcc-12.2 
 +                  - anaconda/2024.02
 </code> </code>
  
Line 407: Line 415:
 [/home/1006/.valet/my-sci-app.vpkg_yaml] [/home/1006/.valet/my-sci-app.vpkg_yaml]
 my-sci-app  Some scientific app project in Python my-sci-app  Some scientific app project in Python
-* 20240307  environment built Mar 7, 2024+* 20250125  environment built Jan 21, 2025
 </code> </code>
  
Line 413: Line 421:
  
 <code bash> <code bash>
-$ vpkg_require my-sci-app/20240307+$ vpkg_require my-sci-app/20250121
 Adding dependency `gcc/12.2.0` to your environment Adding dependency `gcc/12.2.0` to your environment
-Adding dependency `intel-oneapi/2024.0.1.46` to your environment +Adding dependency `ucx/1.13.1` to your environment 
-Adding package `my-sci-app/20240307` to your environment +Adding dependency `openmpi/4.1.5:gcc-12.2` to your environment 
-(/home/1006/conda-envs/my-sci-app/20240307)$ which python3 +Adding dependency `anaconda/2024.02:python3` to your environment 
-~/conda-envs/my-sci-app/20240305/bin/python3 +Adding package `my-sci-app/20250121` to your environment 
-(/home/1006/conda-envs/my-sci-app/20240305)$ pip list | grep mpi4py +(/home/1006/conda-envs/my-sci-app/20250121)$ which python3 
-mpi4py             3.1.5 +~/conda-envs/my-sci-app/20250121/bin/python3 
-(/home/1006/conda-envs/my-sci-app/20240305)$ which mpirun +(/home/1006/conda-envs/my-sci-app/20250121)$ which mpirun 
-/opt/shared/intel-oneapi/2024.0.1.46/mpi/2021.11/bin/mpirun+/opt/shared/openmpi/4.1.5-gcc-12.2/bin/mpirun
 </code> </code>
  
  • technical/recipes/mpi4py-in-virtualenv.1709848303.txt.gz
  • Last modified: 2024-03-07 16:51
  • by anita