Singularity on Caviness

There are several ways to get a Singularity container onto Caviness, but here are the two to consider using if reproducibility is important.

• You can build a Singularity container from a Docker or Singularity Hub on Caviness.
• You can build a container on your local system and copy it to Caviness.

Caviness does not support Docker, but you can build a Singularity container from an existing Docker container or from a Singularity Hub container regsitry. Starting on the head node, load the Singularity software into your environment using VALET, and build your container.

$ vpkg_require singularity
$ singularity build tensorflow.simg docker://tensorflow/tensorflow

$ vpkg_require singularity
$ singularity build hello-world.simg shub://vsoch/hello-worldw

More details on this can be found in the Singularity User Guide.

One example of a container registry is https://biocontainers.pro/registry/#/.

You can build your Singularity container on your local system, installing the singularity program if you need to. For more information on how to do this, see the Singularity web site. Copy your container to Caviness using the usual file transfer methods. See the Transferring files to/from Caviness for more information.

Once your container is on Caviness you can run it. Containers must run on Caviness' compute nodes, not on the head node. Remember you must specify a workgroup before running any jobs on Caviness, then use either the salloc or sbatch command to get access to a compute node. See the Running Jobs on Caviness for more information on SLURM and using Caviness' compute nodes.

vpkg_require singularity

Then you can use the singularity commands to execute your container, like the example below.

[traine@login01 ~]$ workgroup -g it_css
[(it_css:traine)@login01 ~]$ salloc --ntasks=1 --cpus-per-task=4
salloc: Granted job allocation 844
salloc: Waiting for resource configuration
salloc: Nodes r00n45 are ready for job
[traine@r00n45 ~]$ vpkg_require singularity
Adding package `singularity/2.5.1` to your environment
[traine@r00n45 ~]$ singularity shell tensorflow.simg 
Singularity: Invoking an interactive shell within container...

Singularity tensorflow.simg:~> tensorboard --help
usage: tensorboard [-h] [--logdir PATH] [--host ADDR] [--port PORT]
                   [--purge_orphaned_data BOOL] [--reload_interval SECONDS]
                   [--db URI] [--inspect] [--tag TAG] [--event_file PATH]
                   [--path_prefix PATH] [--window_title TEXT]
                   [--max_reload_threads COUNT]
                   [--samples_per_plugin SAMPLES_PER_PLUGIN]
                   [--master_tpu_unsecure_channel ADDR]
                   [--debugger_data_server_grpc_port PORT]
                   [--debugger_port PORT]

TensorBoard is a suite of web applications for inspectinng and understanding
your TensorFlow runs and graphs. https://github.com/tensorflow/tensorboard

Singularity tensorflow.simg:~> grep Cpus_allowed_list /proc/$$/status
Cpus_allowed_list:	0-3
Singularity tensorflow.simg:~> exit
exit
[traine@r00n45 ~]$

If you experience the error below after starting a particular shell container, then it is likely due to your current locale not being supported in this particular container's shell.

[traine@r00n45 ~]$ vpkg_require singularity
Adding package `singularity/2.5.1` to your environment
[traine@r00n45 ~]$ singularity shell tensorflow.simg
Singularity: Invoking an interactive shell within container...
 
Singularity tensorflow.simg:~> tensorboard --help
Traceback (most recent call last):
  File "/usr/local/bin/tensorboard", line 11, in <module>
    sys.exit(run_main())
  File "/usr/local/lib/python2.7/dist-packages/tensorboard/main.py", line 48, in run_main
    program.setup_environment()
  File "/usr/local/lib/python2.7/dist-packages/tensorboard/program.py", line 57, in setup_environment
    util.setup_logging()
  File "/usr/local/lib/python2.7/dist-packages/tensorboard/util.py", line 50, in setup_logging
    locale.setlocale(locale.LC_ALL, '')
  File "/usr/lib/python2.7/locale.py", line 581, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting

Once you start the container shell, check which locale's are supported by using the locale -a command:

Singularity tensorflow.simg:~> locale -a
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_COLLATE to default locale: No such file or directory
C
C.UTF-8
POSIX

Of course in this particular case, the locale setting for account traine is defined as en_US.UTF-8 which is not supported in this shell. To fix the situation, simply set the LANG locale to one that is supported before starting the container shell.

[traine@r00n45 ~]$ vpkg_require singularity
Adding package `singularity/2.5.1` to your environment
[tainne@r00n45 ~]$ LANG=C.UTF-8 singularity shell tensorflow.simg
Singularity: Invoking an interactive shell within container...
 
Singularity tensorflow.simg:~> tensorboard --help
usage: tensorboard [-h] [--logdir PATH] [--host ADDR] [--port PORT]
                   [--purge_orphaned_data BOOL] [--reload_interval SECONDS]
                   [--db URI] [--inspect] [--tag TAG] [--event_file PATH]
                   [--path_prefix PATH] [--window_title TEXT]
                   [--max_reload_threads COUNT]
                   [--samples_per_plugin SAMPLES_PER_PLUGIN]
                   [--master_tpu_unsecure_channel ADDR]
                   [--debugger_data_server_grpc_port PORT]
                   [--debugger_port PORT]
 
TensorBoard is a suite of web applications for inspectinng and understanding
your TensorFlow runs and graphs. https://github.com/tensorflow/tensorboard
 
[[ lots of additional information displayed for optional arguments
 
Singularity tensorflow.simg:~> exit
exit
[traine@r00n45 ~]$

Some common commands are listed here. For more information about Singularity, see the Singularity web site.