technical:recipes:mpi4py-in-virtualenv

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
technical:recipes:mpi4py-in-virtualenv [2024-11-22 13:33] – [Create the Virtual Environment] anitatechnical:recipes:mpi4py-in-virtualenv [2025-01-31 15:29] (current) – [VALET Package Definition] anita
Line 15: Line 15:
  
 <WRAP center round info 60%> <WRAP center round info 60%>
-On Caviness and DARWIN we would likely choose the Intel distribution selecting a Python 3 version over Anaconda (e.g. ''vpkg_versions intel-oneapi'' or ''intel-python'') since it automatically enables Intel's distribution channel.  That channel includes Numpy built against the Intel MKL library, for example, and other highly-optimized variants of computationally-intensive Python components.+Due to recent announcements regarding Anaconda, and Intel dropping their distribution channel, any documentation referring to Intel's channel will need to be updated.
  
-As of November 2020, the majority of packages populating the Intel channel require baseline operating system libraries (like ''glibc'') newer than what Farber provides:  a clear example of the binary compatibility issues that are present in conda software distribution.+Please use ''conda-forge'' channel for installations.
 </WRAP> </WRAP>
  
Line 213: Line 213:
 ==== Building mpi4py ==== ==== Building mpi4py ====
  
-With the new virtual environment activated, we can now build mpi4py against the local Open MPI library we added to the shell environment. Due to Anaconda trying to use a version of ''ld'' as part of the virtual environment in lieu of the system ''ld'', you need to change the permissions to allow the compile to work building mpi4y.+With the new virtual environment activated, we can now build mpi4py against the local Open MPI library we added to the shell environment. Due to Anaconda trying to use a version of ''ld'' as part of the virtual environment in lieu of the system ''ld'', you need to change the permissions to allow the compile to work properly.
  
 <code base> <code base>
Line 263: Line 263:
               description: environment built Nov 2, 2020               description: environment built Nov 2, 2020
               dependencies:               dependencies:
-                  - openmpi/4.0.2 +                  - openmpi/4.1.4:gcc-12.1.0 
-                  - intel-python/2020u2:python3+                  - anaconda/2024.02
 </code> </code>
  
Line 285: Line 285:
 <code bash> <code bash>
 $ vpkg_require my-sci-app/20201102 $ vpkg_require my-sci-app/20201102
-Adding dependency `libfabric/1.9.0` to your environment +Adding dependency `libfabric/1.13.2` to your environment 
-Adding dependency `openmpi/4.0.2` to your environment +Adding dependency `binutils/2.35` to your environment 
-Adding dependency `intel-python/2020u2:python3` to your environment+Adding dependency `gcc/12.1.0` to your environment 
 +Adding package `openmpi/4.1.4:gcc-12.1.0` to your environment 
 +Adding package `anaconda/2024.02` to your environment
 Adding package `my-sci-app/20201102` to your environment Adding package `my-sci-app/20201102` to your environment
 (/home/1001/conda-envs/my-sci-app/20201102)$ which python3 (/home/1001/conda-envs/my-sci-app/20201102)$ which python3
 ~/conda-envs/my-sci-app/20201102/bin/python3 ~/conda-envs/my-sci-app/20201102/bin/python3
 (/home/1001/conda-envs/my-sci-app/20201102)$ pip list | grep mpi4py (/home/1001/conda-envs/my-sci-app/20201102)$ pip list | grep mpi4py
-mpi4py      3.0.3+mpi4py      4.0.1
 $ which mpirun $ which mpirun
-/opt/shared/openmpi/4.0.2/bin/mpirun+/opt/shared/openmpi/4.1.4:gcc-12.1.0/bin/mpirun
 </code> </code>
  
Line 302: Line 304:
  
 <code bash> <code bash>
-$ vpkg_require openmpi/5.0.2:intel-oneapi-2024 intel-oneapi/2024+$ vpkg_require openmpi/4.1.5:gcc-12.2 anaconda/2024.02
 Adding dependency `gcc/12.2.0` to your environment Adding dependency `gcc/12.2.0` to your environment
-Adding dependency `intel-oneapi/2024.0.1.46` to your environment 
 Adding dependency `ucx/1.13.1` to your environment Adding dependency `ucx/1.13.1` to your environment
-Adding package `openmpi/5.0.2:intel-oneapi-2024` to your environment+Adding package `openmpi/4.1.5:gcc-12.2` to your environment 
 +Adding package `anaconda/2024.02:python3` to your environment
 </code> </code>
  
Line 314: Line 316:
  
 <code bash> <code bash>
-$ conda create --prefix ${HOME}/conda-envs/my-sci-app/20240307 --channel intel --override-channels python'=>3.9' numpy scipy+$ conda create --prefix ${HOME}/conda-envs/my-sci-app/20250121 --channel defaults --override-channels python'=>3.7' numpy scipy
 Collecting package metadata (current_repodata.json): done Collecting package metadata (current_repodata.json): done
 Solving environment: done Solving environment: done
Line 323: Line 325:
 # To activate this environment, use # To activate this environment, use
 # #
-#     $ conda activate /home/1006/conda-envs/my-sci-app/20240307+#     $ conda activate /home/1006/conda-envs/my-sci-app/20250121
 # #
 # To deactivate an active environment, use # To deactivate an active environment, use
Line 334: Line 336:
  
 <code bash> <code bash>
-$ conda activate /home/1006/conda-envs/my-sci-app/20240307 +$ conda activate /home/1006/conda-envs/my-sci-app/20250121 
-(/home/1006/conda-envs/my-sci-app/20240307)$+(/home/1006/conda-envs/my-sci-app/20250121)$
 </code> </code>
  
Line 343: Line 345:
  
 <code base> <code base>
-(/home/1006/conda-envs/my-sci-app/20240307)$ pip install --no-binary :all: --compile mpi4py+(/home/1006/conda-envs/my-sci-app/20250121)$ chmod 000 /home/1001/conda-envs/my-sci-app/20201102/compiler_compat/ld 
 +(/home/1006/conda-envs/my-sci-app/20250121)$ pip install --no-binary :all: --compile mpi4py 
 +$ pip install --no-binary :all: --compile mpi4py
 Collecting mpi4py Collecting mpi4py
-  Downloading mpi4py-3.1.5.tar.gz (2.5 MB) +  Downloading mpi4py-4.0.1.tar.gz (466 kB)
-     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 16.7 MB/s eta 0:00:00+
   Installing build dependencies ... done   Installing build dependencies ... done
   Getting requirements to build wheel ... done   Getting requirements to build wheel ... done
 +  Installing backend dependencies ... done
   Preparing metadata (pyproject.toml) ... done   Preparing metadata (pyproject.toml) ... done
 Building wheels for collected packages: mpi4py Building wheels for collected packages: mpi4py
   Building wheel for mpi4py (pyproject.toml) ... done   Building wheel for mpi4py (pyproject.toml) ... done
-  Created wheel for mpi4py: filename=mpi4py-3.1.5-cp310-cp310-linux_x86_64.whl size=634821 sha256=78a58c10acd22b3cf2ebf9e73b445d6775ac29f3f59c37e63bd16e27b7467ba2 +  Created wheel for mpi4py: filename=mpi4py-4.0.1-cp313-cp313-linux_x86_64.whl size=997834 sha256=b09b4fe26c8aa940bdcbdb512960fb73edb9ed9ed698b9455db3e1f3d5b078a5 
-  Stored in directory: /home/1006/.cache/pip/wheels/18/2b/7f/c852523089e9182b45fca50ff56f49a51eeb6284fd25a66713+  Stored in directory: /home/1006/.cache/pip/wheels/27/79/62/f500b54e8b8ce5f5e54e7b84e8695938988ca274117d39983b
 Successfully built mpi4py Successfully built mpi4py
 Installing collected packages: mpi4py Installing collected packages: mpi4py
-Successfully installed mpi4py-3.1.5+Successfully installed mpi4py-4.0.1
 </code> </code>
  
-The ''--no-binary :all:'' flag prohibits the installation of any packages that include binary components, effectively forcing a rebuild of mpi4py from source.  The ''--compile'' flag pre-processes all Python scripts in the mpi4py package (versus allowing them to be processed and cached later).  The environment now includes support for mpi4py linked against the ''''openmpi/5.0.2:intel-oneapi-2024'''' library on DARWIN:+The ''--no-binary :all:'' flag prohibits the installation of any packages that include binary components, effectively forcing a rebuild of mpi4py from source.  The ''--compile'' flag pre-processes all Python scripts in the mpi4py package (versus allowing them to be processed and cached later).  The environment now includes support for mpi4py linked against the ''''openmpi/4.1.4:gcc-12.2.0'''' library on DARWIN:
  
 <code bash> <code bash>
-(/home/1006/conda-envs/my-sci-app/20240307)$ pip list | grep mpi4py +(/home/1006/conda-envs/my-sci-app/20250121)$ pip list | grep mpi4py 
-mpi4py             3.1.5+mpi4py      4.0.1
 </code> </code>
  
Line 393: Line 397:
           success: 0           success: 0
     versions:     versions:
-          "20240307": +          "20250121": 
-              description: environment built Mar 7, 2024+              description: environment built Jan 21, 2025
               dependencies:               dependencies:
-                  - openmpi/5.0.2:intel-oneapi-2024 +                  - openmpi/4.1.5:gcc-12.2 
-                  - intel-oneapi/2024+                  - anaconda/2024.02
 </code> </code>
  
Line 411: Line 415:
 [/home/1006/.valet/my-sci-app.vpkg_yaml] [/home/1006/.valet/my-sci-app.vpkg_yaml]
 my-sci-app  Some scientific app project in Python my-sci-app  Some scientific app project in Python
-* 20240307  environment built Mar 7, 2024+* 20250125  environment built Jan 21, 2025
 </code> </code>
  
Line 417: Line 421:
  
 <code bash> <code bash>
-$ vpkg_require my-sci-app/20240307+$ vpkg_require my-sci-app/20250121
 Adding dependency `gcc/12.2.0` to your environment Adding dependency `gcc/12.2.0` to your environment
-Adding dependency `intel-oneapi/2024.0.1.46` to your environment 
 Adding dependency `ucx/1.13.1` to your environment Adding dependency `ucx/1.13.1` to your environment
-Adding dependency `openmpi/5.0.2:intel-oneapi-2024` to your environment +Adding dependency `openmpi/4.1.5:gcc-12.2` to your environment 
-Adding package `my-sci-app/20240307` to your environment +Adding dependency `anaconda/2024.02:python3` to your environment 
-(/home/1006/conda-envs/my-sci-app/20240307)$ which python3 +Adding package `my-sci-app/20250121` to your environment 
-~/conda-envs/my-sci-app/20240305/bin/python3 +(/home/1006/conda-envs/my-sci-app/20250121)$ which python3 
-(/home/1006/conda-envs/my-sci-app/20240305)$ pip list | grep mpi4py +~/conda-envs/my-sci-app/20250121/bin/python3 
-mpi4py             3.1.5 +(/home/1006/conda-envs/my-sci-app/20250121)$ which mpirun 
-(/home/1006/conda-envs/my-sci-app/20240305)$ which mpirun +/opt/shared/openmpi/4.1.5-gcc-12.2/bin/mpirun
-/opt/shared/openmpi/5.0.2-intel-oneapi-2024/bin/mpirun+
 </code> </code>
  
  • technical/recipes/mpi4py-in-virtualenv.1732300380.txt.gz
  • Last modified: 2024-11-22 13:33
  • by anita