| Both sides previous revision Previous revision | Last revisionBoth sides next revision |
| software:python:python [2021-09-30 14:14] – [Details by cluster] anita | software:python:python [2024-02-20 09:48] – [Python] anita |
|---|
| | ====== Python ====== |
| |
| | [[https://www.python.org/|Python]] is an interpreted, object-oriented, high-level programming language with dynamic semantics. Its high-level built in data structures, combined with dynamic typing and dynamic binding, make it very attractive for Rapid Application Development, as well as for use as a scripting or glue language to connect existing components together. |
| | |
| | There are various Python versions (platforms) and packages installed on each cluster. Please use the VALET command ''vpkg_list | grep python'' on the cluster you are using to see all related Python software installations. [[software:anaconda:anaconda|Anaconda]] is a popular Python platform that may be more suitable based on the documentation provided for your software installation use or requirements. [[software:mpi4py:mpi4py|Mpi4py]] is also another Python platform with bindings of the Message Passing Interface (MPI) standard for the Python programming language allowing any Python program to exploit multiple processors. |
| | |
| | Running Python jobs on our clusters, in particularly, machine learning and/or multiprocessing (parallel) jobs will likely use shared memory on each node assigned to the job. Unfortunately, shared memory (''/dev/shm'') cannot be easily reserved or cleared as part of your job submission via the job scheduler on each cluster, as a result, you may receive the following error |
| | |
| | <note important>OSError: [Errno 28] No space left on device</note> |
| | |
| | It has been suggested setting the environment variable ''JOBLIB_TEMP_DIR=${TMPDIR}'' as part of your job submission or defining it in your Python code for multiprocessing something like this |
| | |
| | <code bash> |
| | from multiprocessing import process |
| | process . current_process (). _config [ 'tempdir' ] = '${TMPDIR}' |
| | </code> |
| | |
| | will use the local scratch storage on the node versus shared memory and avoid the error described above. |
| | |
| | ===== Details by cluster ===== |
| | |
| | * Caviness |
| | * [[software:anaconda:caviness | Anaconda]] |
| | * [[software:mpi4py:caviness| Mpi4py]] |
| | * DARWIN |
| | * [[software:anaconda:darwin | Anaconda]] |
| | * [[technical:recipes:mpi4py-in-virtualenv|Python Virtual Environments with mpi4py]] |
| | * Farber |
| | * [[software:anaconda:farber | Anaconda]] |
| | * [[software:mpi4py:farber| Mpi4py]] |
| | |