Python is an interpreted, object-oriented, high-level programming language with dynamic semantics. Its high-level built in data structures, combined with dynamic typing and dynamic binding, make it very attractive for Rapid Application Development, as well as for use as a scripting or glue language to connect existing components together.

There are various Python versions (platforms) and packages installed on each cluster. Please use the VALET command vpkg_list | grep python on the cluster you are using to see all related Python software installations. Anaconda is a popular Python platform that may be more suitable based on the documentation provided for your software installation use or requirements, but the Intel version of Python contains many packages as well as being optimized for Intel processsors. Mpi4py is also another Python platform with bindings of the Message Passing Interface (MPI) standard for the Python programming language allowing any Python program to exploit multiple processors.

Running Python jobs on our clusters, in particularly, machine learning and/or multiprocessing (parallel) jobs will likely use shared memory on each node assigned to the job. Unfortunately, shared memory (/dev/shm) cannot be easily reserved or cleared as part of your job submission via the job scheduler on each cluster, as a result, you may receive the following error

OSError: [Errno 28] No space left on device

It has been suggested setting the environment variable JOBLIB_TEMP_DIR=${TMPDIR} as part of your job submission or defining it in your Python code for multiprocessing something like this

from multiprocessing import process
process . current_process (). _config [ 'tempdir' ] = '${TMPDIR}'

will use the local scratch storage on the node versus shared memory and avoid the error described above.

  • software/python/python.txt
  • Last modified: 2024-02-20 09:49
  • by anita