This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revisionBoth sides next revision |
software:python:python [2020-07-10 14:30] – anita | software:python:python [2021-09-30 14:14] – [Details by cluster] anita |
---|
====== Python ====== | |
| |
[[https://www.python.org/|Python]] is an interpreted, object-oriented, high-level programming language with dynamic semantics. Its high-level built in data structures, combined with dynamic typing and dynamic binding, make it very attractive for Rapid Application Development, as well as for use as a scripting or glue language to connect existing components together. | |
| |
There are various Python versions (platforms) and packages installed on each cluster. Please use the VALET command ''vpkg_list | grep python'' the cluster you are using to see all related Python software installation. [[software:anaconda:anaconda|Anaconda]] is a popular Python platform that may be more suitable based on the documentation provided for your software installation use or requirements. [[software:mpi4py:mpi4py|Mpi4py]] is also another Python platform with bindings of the Message Passing Interface (MPI) standard for the Python programming language allowing any Python program to exploit multiple processors. | |
| |
Running Python jobs on our clusters, in particularly, machine learning and/or multiprocessing (parallel) jobs will likely use shared memory on each node assigned to the job. Unfortunately, shared memory (''/dev/shm'') cannot be easily reserved or cleared as part of your job submission via the job scheduler on each cluster, as a result, you may receive the following error | |
| |
<note important>OSError: [Errno 28] No space left on device</note> | |
| |
It has been suggested setting the environment variable ''JOBLIB_TEMP_DIR=${TMPDIR}'' as part of your job submission or defining it in your python code for multiprocessing something like this | |
| |
<code bash> | |
from multiprocessing import process | |
process . current_process (). _config [ 'tempdir' ] = '${TMPDIR}' | |
</code> | |
| |
will use the local scratch storage on the node versus shared memory and avoid the error described above. | |