Differences

This shows you the differences between two versions of the page.

--- software:python:python [2020-07-09 18:08] – anita
+++ software:python:python [2024-02-20 09:49] (current) – [Python] anita
@@ Line 3: / Line 3: @@
 [[https://www.python.org/|Python]] is an interpreted, object-oriented, high-level programming language with dynamic semantics. Its high-level built in data structures, combined with dynamic typing and dynamic binding, make it very attractive for Rapid Application Development, as well as for use as a scripting or glue language to connect existing components together.
-There are various Python versions (platforms) and packages installed on each cluster. Please use the VALET command ''vpkg_list | grep python'' to see all related Python software installation.  [[software:anaconda:anaconda|Anaconda]] is a popular Python platform that may be more suitable based on the documentation provided for your software installation requirements. [[software:mpi4py:mpi4py|Mpi4py]] is also another Python platform that allowing any Python program to exploit multiple processors.
+There are various Python versions (platforms) and packages installed on each cluster. Please use the VALET command ''vpkg_list | grep python'' on the cluster you are using to see all related Python software installations.  [[software:anaconda:anaconda|Anaconda]] is a popular Python platform that may be more suitable based on the documentation provided for your software installation use or requirements, but the Intel version of Python contains many packages as well as being optimized for Intel processsors. [[software:mpi4py:mpi4py|Mpi4py]] is also another Python platform with bindings of the Message Passing Interface (MPI) standard for the Python programming language allowing any Python program to exploit multiple processors.
-Running Python jobs on our clusters, in particularly, machine learning and/or multiprocessing (parallel) will likely use shared memory on each node assigned to the job.  Unfortunately shared memory (''/dev/shm'') cannot be easily reserved or cleared as part of your job submission, as a result, you may receive the following error
+Running Python jobs on our clusters, in particularly, machine learning and/or multiprocessing (parallel) jobs will likely use shared memory on each node assigned to the job.  Unfortunately, shared memory (''/dev/shm'') cannot be easily reserved or cleared as part of your job submission via the job scheduler on each cluster, as a result, you may receive the following error
 <note important>OSError: [Errno 28] No space left on device</note>
-It has been suggested setting the environment variable ''JOBLIB_TEMP_DIR=${TMPDIR}'' will use the local scratch storage on the node versus shared memory and avoid the above error.
+It has been suggested setting the environment variable ''JOBLIB_TEMP_DIR=${TMPDIR}'' as part of your job submission or defining it in your Python code for multiprocessing something like this
+<code bash>
+from multiprocessing import process
+process . current_process (). _config [ 'tempdir' ] = '${TMPDIR}'
+</code>
+will use the local scratch storage on the node versus shared memory and avoid the error described above.
+===== Details by cluster =====
+  * Caviness
+      * [[software:anaconda:caviness | Anaconda]]
+      * [[software:mpi4py:caviness| Mpi4py]]
+  * DARWIN
+      * [[software:anaconda:darwin | Anaconda]]
+      * [[technical:recipes:mpi4py-in-virtualenv|Python Virtual Environments with mpi4py]]
+  * Farber
+      * [[software:anaconda:farber | Anaconda]]
+      * [[software:mpi4py:farber| Mpi4py]]