software:singularity:caviness

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
software:singularity:caviness [2018-08-10 23:23] – [Inside your interactive session or your batch job you must first issue the command] anitasoftware:singularity:caviness [2024-08-27 17:51] (current) – anita
Line 1: Line 1:
 +======= Apptainer (formerly Singularity) on Caviness ======
 +
 +===== Build a Singularity container =====
 +
 +There are several ways to get a Singularity container onto Caviness, but here are the two to consider using if reproducibility is important.
 +
 +  * You can build a Singularity container from a Docker or Singularity Hub on Caviness.
 +  * You can build a container on your local system and copy it to Caviness.
 +
 +==== Build a Singularity container from Docker or Singularity Hub Registry ====
 +
 +Caviness does not support Docker, but you can build a Singularity container from an existing Docker container or from a Singularity Hub container regsitry. Starting on the head node, load the Singularity software into your environment using VALET, and build your container.
 +
 +=== Docker Hub example ===
 +
 +<code>
 +$ vpkg_require singularity
 +$ singularity build tensorflow.simg docker://tensorflow/tensorflow
 +</code>
 +
 +=== Singularity Hub example ===
 +
 +<code>
 +$ vpkg_require singularity
 +$ singularity build hello-world.simg shub://vsoch/hello-worldw
 +</code>
 +
 +More details on this can be found in the [[http://singularity.lbl.gov/quickstart|Singularity User Guide]].
 +
 +One example of a container registry is [[https://biocontainers.pro/registry/#/]].
 +
 +==== Build a container on your local system ====
 +
 +You can build your Singularity container on your local system, installing the singularity program if you need to.  For more information on how to do this, see the [[http://singularity.lbl.gov/quickstart|Singularity web site]].   Copy your container to Caviness using the usual file transfer methods. See the [[abstract:caviness:transfer|Transferring files to/from Caviness]] for more information.
 +
 +===== Execute your Singularity container through SLURM =====
 +
 +Once your container is on Caviness you can run it. Containers must run on Caviness' compute nodes, not on the head node. Remember you must specify a **[[abstract:caviness:app_dev:compute_env#using-workgroup-and-directories|workgroup]]** before running any jobs on Caviness, then use either the ''salloc'' or ''sbatch'' command to get access to a compute node.  See the [[abstract:caviness:runjobs:runjobs|Running Jobs on Caviness]] for more information on SLURM and using Caviness' compute nodes.
 +
 +==== Inside your interactive session or your batch job you must first issue the command ====
 +
 +<code>
 +vpkg_require singularity
 +</code>
 +
 +Then you can use the singularity commands to execute your container, like the example below.
 +
 +<code>
 +[traine@login01 ~]$ workgroup -g it_css
 +[(it_css:traine)@login01 ~]$ salloc --ntasks=1 --cpus-per-task=4
 +salloc: Granted job allocation 844
 +salloc: Waiting for resource configuration
 +salloc: Nodes r00n45 are ready for job
 +[traine@r00n45 ~]$ vpkg_require singularity
 +Adding package `singularity/2.5.1` to your environment
 +[traine@r00n45 ~]$ singularity shell tensorflow.simg 
 +Singularity: Invoking an interactive shell within container...
 +
 +Singularity tensorflow.simg:~> tensorboard --help
 +usage: tensorboard [-h] [--logdir PATH] [--host ADDR] [--port PORT]
 +                   [--purge_orphaned_data BOOL] [--reload_interval SECONDS]
 +                   [--db URI] [--inspect] [--tag TAG] [--event_file PATH]
 +                   [--path_prefix PATH] [--window_title TEXT]
 +                   [--max_reload_threads COUNT]
 +                   [--samples_per_plugin SAMPLES_PER_PLUGIN]
 +                   [--master_tpu_unsecure_channel ADDR]
 +                   [--debugger_data_server_grpc_port PORT]
 +                   [--debugger_port PORT]
 +
 +TensorBoard is a suite of web applications for inspectinng and understanding
 +your TensorFlow runs and graphs. https://github.com/tensorflow/tensorboard
 +
 +Singularity tensorflow.simg:~> grep Cpus_allowed_list /proc/$$/status
 +Cpus_allowed_list: 0-3
 +Singularity tensorflow.simg:~> exit
 +exit
 +[traine@r00n45 ~]$
 +</code>
 +
 +==== Troubleshooting ====
 +
 +=== Locale error ===
 +
 +If you experience the error below after starting a particular shell container, then it is likely due to your current locale not being supported in this particular container's shell.  
 +
 +<code bash>
 +[traine@r00n45 ~]$ vpkg_require singularity
 +Adding package `singularity/2.5.1` to your environment
 +[traine@r00n45 ~]$ singularity shell tensorflow.simg
 +Singularity: Invoking an interactive shell within container...
 +
 +Singularity tensorflow.simg:~> tensorboard --help
 +Traceback (most recent call last):
 +  File "/usr/local/bin/tensorboard", line 11, in <module>
 +    sys.exit(run_main())
 +  File "/usr/local/lib/python2.7/dist-packages/tensorboard/main.py", line 48, in run_main
 +    program.setup_environment()
 +  File "/usr/local/lib/python2.7/dist-packages/tensorboard/program.py", line 57, in setup_environment
 +    util.setup_logging()
 +  File "/usr/local/lib/python2.7/dist-packages/tensorboard/util.py", line 50, in setup_logging
 +    locale.setlocale(locale.LC_ALL, '')
 +  File "/usr/lib/python2.7/locale.py", line 581, in setlocale
 +    return _setlocale(category, locale)
 +locale.Error: unsupported locale setting
 +</code>
 +
 +Once you start the container shell, check which locale's are supported by using the ''locale -a'' command:
 +
 +<code bash>
 +Singularity tensorflow.simg:~> locale -a
 +locale: Cannot set LC_CTYPE to default locale: No such file or directory
 +locale: Cannot set LC_MESSAGES to default locale: No such file or directory
 +locale: Cannot set LC_COLLATE to default locale: No such file or directory
 +C
 +C.UTF-8
 +POSIX
 +</code>
 +
 +Of course in this particular case, the locale setting for account ''traine'' is defined as ''en_US.UTF-8'' which is not supported in this shell.  To fix the situation, simply set the ''LANG'' locale to one that is supported before starting the container shell.
 +
 +<code bash>
 +[traine@r00n45 ~]$ vpkg_require singularity
 +Adding package `singularity/2.5.1` to your environment
 +[tainne@r00n45 ~]$ LANG=C.UTF-8 singularity shell tensorflow.simg
 +Singularity: Invoking an interactive shell within container...
 +
 +Singularity tensorflow.simg:~> tensorboard --help
 +usage: tensorboard [-h] [--logdir PATH] [--host ADDR] [--port PORT]
 +                   [--purge_orphaned_data BOOL] [--reload_interval SECONDS]
 +                   [--db URI] [--inspect] [--tag TAG] [--event_file PATH]
 +                   [--path_prefix PATH] [--window_title TEXT]
 +                   [--max_reload_threads COUNT]
 +                   [--samples_per_plugin SAMPLES_PER_PLUGIN]
 +                   [--master_tpu_unsecure_channel ADDR]
 +                   [--debugger_data_server_grpc_port PORT]
 +                   [--debugger_port PORT]
 +
 +TensorBoard is a suite of web applications for inspectinng and understanding
 +your TensorFlow runs and graphs. https://github.com/tensorflow/tensorboard
 +
 +[[ lots of additional information displayed for optional arguments
 +
 +Singularity tensorflow.simg:~> exit
 +exit
 +[traine@r00n45 ~]$
 +</code>
 +
 +==== Common Singularity commands ====
 +
 +Some common commands are listed here. For more information about Singularity, see the [[http://singularity.lbl.gov/quickstart|Singularity web site]]. 
 +
 +| ''shell'' | Start a shell within your container using the operating system you have set up your container to use.|
 +| ''exec'' | Run a single command within your container.|
 +| ''run'' | Run a recipe script you have set up within your container. Using a recipe script forces users of your container to use a pre-established workflow.|
 +| ''help'' | Provides help on Singularity.|