Various members of the CEMAC team attended the ARCHER 2 containers workshop in July 2021.

Here's a quick start guide from that tutorial

• Installation of docker and sigularity are out lined here

The course outline is found here

Why use containers? - main point is reproducibility with software i.e.

• time goes by and dependencies break
• missing bits of documentation
• systems no longer exist with software stack
• cloud system no longer have access to

To not have to run as sudo look here

Once docker is installed

Start the docker engine

dockerd

check commands work

docker --version
docker container ls
docker container --help
docker container ls --help  
docker build --help

docker container ls
docker pull hello-world
docker container ls
docker run hello-world

You don't have to bother with pull command docker run will do this if the container isn't found locally e.g.

docker run alpine

we can get the container to print its os info or print hello world

docker run alpine cat /etc/os-release
docker run alpine echo 'Hello World'

note the speed of this compared to a VM

we can also run this interactively

docker run -it alpine sh

you can run various linux commands now and and run exit to stop

Owned by docker team:

# docker pull CONTAINER:TAG
docker pull python:3.8

Docker hub community space (check the containers are legit before downloading)

# docker pull OWNER/CONTAINER:TAG
docker pull rocker/tidyverse:3.6.1

# running
docker ps
# run recently and get list of recent exited containers
docker container ls --all

# single container
docker container rm <container ID>
# tidy all
docker container prune

Now we can remove images (if you've not done the steps above you may get an error)

# get image id
docker image ls
docker image rm <image id>

• Create Dockerfile

docker build -t <username>/docker_example Docker_example/

docker will build any Dockerfile it finds in the directory specified

now you can run:

docker image ls
docker run <username>/docker_example echo "Hello World"

docker login
docker tag <username>docker_example <username>/docker_test_example:v1
docker push <username>/docker_test_example:v1

• keep simple software stack
• use linux flavours like ubuntu or centos

docker run -v <pathtodirhost>:/temp <username>/docker_example python3 <pathtodircontainer>/sum.py

sum.py takes input vars and sums them so you can add numbers to the end of the above script to alter the output

This Dockerfile will build the container with the sum function as the default fuction with some default input options

This Dockerfile will build the container with the sum function added to the path but you can also use other functions and commands easily

docker build -t <username>/complex_docker_example:v1 <pathtodirwithdockerfile>

This example COPY local files but you can also get files via wget or git

RUN git clone https://github.com/<username>/<repo>
RUN wget <url>

This is mostly explained in the course documentation here

Singularity is availble on archer2 and arc4 you might need to module load it in

on archer2

singularity --version

shows its working.

No need to start daemone like with docker. I a test dir

singularity pull hello-world.sif shub://vsoch/hello-world
singularity run hello-world.sif
singularity inspect -r hello-world.sif

rm hello-world.sif
singularity pull hello-world.sif shub://vsoch/hello-world
singularity cache list
singularity cache list -v
singularity cache clean

exec allows us to run the non default command

singularity exec hello-world.sif /bin/echo Hello World!
singularity exec hello-world.sif /bin/date

enter the container with the shell command

singularity shell hello-world.sif

running whoami will display your username rather than root and will show the current directory, often bind mounting to home dir - works for your work directory as well

you won't be able to access the other files on the system e.g. other users directory

you can create files on the host system with in the container

singularity shell -B /work/z19/shared hello-world.sif

you can pull docker images and convert to sif files

singularity pull python-3.9.6.sif docker://python:3.9.6-slim-buster

You need route access to create SIF files

There are three different options for accessing a suitable environment to undertake the material in this part of the course:

1. Run Singularity from within a Docker container - this will enable you to have the required privileges to build images
2. Install Singularity locally on a system where you have administrative access
3. Use Singularity on a system where it is already pre-installed and you have administrative (root) access

sigularity Installation on a local machine requires lots of dependencies so this tutorial uses a docker singularity contaier and aliases it as singularity

alias singularity='docker run --privileged -v ${PWD}:/home/singularity quay.io/singularity/singularity:v3.5.3-slim'

NB might need sudo in the alias on linux