Singularity

Singularity as an alternative container tool#

Singularity is a container software alternative to Docker. It was originally developed by researchers at Lawrence Berkeley National Laboratory with focus on security, scientific software, and HPC clusters. One of the ways in which Singularity is more suitable for HPC is that it very actively restricts permissions so that you do not gain access to additional resources while inside the container.

Here we give a brief introduction to Singularity and specifically how it can be used on HPC clusters.

If you want to read more, here are some additional resources:

• Singularity docs
• HPC Singularity user guide

More useful commands in Singularity#

Build a container

singularity build Singularity.sif Singularity.def

Run a command from a container

singularity run Singularity.sif echo toto

Executing a script inside the container

singularity exec Singularity.sif

Use a container interactively

singularity run Singularity.sif

you can also define bind points if you need access outside the container. $HOME, /tmp, /proc, /sys, /dev are mount by default but they can configurated.

Many of the Singularity commands such as run, exec , and shell take the --bind command-line option to specify bind paths, in addition to the SINGULARITY_BINDPATH environment variable.

## export SINGULARITY_BINDPATH="/opt,/data:/mnt
## OR -B /opt,/data:/mnt 
singularity run Singularity.sif --bind $DIR

Converting Docker images to Singularity files#

Singularity, unlike Docker, stores images as single files. A Singularity image file is self-contained (no shared layers) and can be moved around and shared like any other file.

While it is possible to define and build Singularity images from scratch, in a manner similar to what you've already learned for Docker, this is not something we will cover here (but feel free to read more about this in e.g. the Singularity docs).

Instead, we will take advantage of the fact that Singularity can convert Docker images to the Singularity Image Format (SIF). This is great if there's a Docker image that you want to use on an HPC cluster where you cannot use Docker.

Let's try to convert the Docker image for this course directly from DockerHub using singularity pull:

singularity pull mrsa_proj.sif docker://nbisweden/workshop-reproducible-research

This should result in a file called mrsa_proj.sif.

Running a singularity image#

In the Docker image we included the code needed for the workflow in the /course directory of the image. These files are of course also available in the Singularity image. However, a Singularity image is read-only (unless using the sandbox feature). This will be a problem if we try to run the workflow within the /course directory, since the workflow will produce files and Snakemake will create a .snakemake directory. Instead, we need to provide the files externally from our host system and simply use the Singularity image as the environment to execute the workflow in (i.e. all the software and dependencies).

In your current working directory (workshop-reproducible-research/tutorials/containers/) the vital MRSA project files are already available (Snakefile, config.yml, code/header.tex and code/supplementary_material.Rmd).

Since Singularity bind mounts the current working directory we can simply execute the workflow and generate the output files using:

singularity run mrsa_proj.sif

This executes the default run command, which is snakemake -rp -c 1 --configfile config.yml (as defined in the original Dockerfile).

The previous step in this tutorial included running the run_qc.sh script, so that part of the workflow has already been run and Snakemake will continue from that automatically without redoing anything. Once completed you should see a bunch of directories and files generated in your current working directory, including the results/ directory containing the final HTML report.

Containers at different platforms#

A common problem with Singularity is that you can only create local builds if you are working on a Linux system, as local builds for MacOS and Windows are currently not supported. This means that you might favour using Docker instead of Singularity, but what happens when you need to use a HPC cluster such as HPC? Docker won't work there, as it requires root privileges, so Singularity is the only solution.