Deployment commons - SeaDataCloud/Documentation GitHub Wiki

These things are useful for deployment, for both master node and service nodes.

Directory organisation

Each service will have its home directory, where its docker-compose.yml lies, and optionally more resources. In Athens, these are placed in /root/<servicename>/, in STFC in /opt/vre/<servicename>/.

Directories that will contain large amounts of data might be in there too, or they might be placed in other locations of the file system, if another disk partition has more space.

Directories that are being used by many services, such as the directories for SSL certs and health check files, could be placed in the same parent directory as the service homes, e.g. /root/HEALTH/ or /opt/vre/HEALTH/.

Install docker and docker-compose

yum install docker

# Note: This below is not the most recent version of docker-compose, please make sure to install up to date
sudo curl -L "https://github.com/docker/compose/releases/download/1.24.0/docker-compose-$(uname -s)-$(uname -m)" -o  /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
docker-compose --version

Running docker-compose stacks

A good way to run a stack is:

# Remove all old containers if desired:
#docker-compose down

# Start the stack in the background (but without attaching to the stdout/stderr of the stack, and without having the console tied to the service)
docker-compose up -d

# Attach to the stack's logs:
docker-compose logs --tail=100 -f

# Now you can safely quit ctrl-C to use the console again without quitting the stack!

To easily do this all the time, you can add this to your ~/.bashrc:

alias doco="docker-compose down && docker-compose up -d && docker-compose logs --tail=100 -f"
alias dolo="docker-compose up -d && docker-compose logs --tail=100 -f"

In general, we start a stack by running docker-compose up inside the stack's home directory (where the docker-compose.yml file is located). The stack starts running and its logs are printed to the screen. If you run ctrl-C, this will stop the stack. (Exiting the console / ssh session does not stop the stack.)

To avoid this, you can run in the -d mode, i.e. in the background (detached mode).

To view the logs of a stack after you have exited the ssh session: docker-compose logs --tail=all -f (for all lines of log since the start of the stack, or --tail=100 for 100 lines of log).

Docker Registry at GRNET

We have a docker registry at GRNET for only SeaDataNet images: registry-sdc.argo.grnet.gr.

To push or pull images, you need to login (ask Themis for credentials).

docker login registry-sdc.argo.grnet.gr

To push an image there, do the following steps.

We appreciate if you could add the date of your build to the image (after a colon). This way we can keep track of which images are deployed where. If you do not add anything, the tag latest will be added, but as new iamges are not automatically pulled, that can lead to confusion (why using latest is not cool, read this: https://medium.com/@mccode/the-misunderstood-docker-tag-latest-af3babfd6375)

# Build you image as usual:
docker build -t myimage:20191231 . 

# Add the registry as an additional tag:
docker tag myimage:20191231 registry-sdc.argo.grnet.gr/myimage:20191231

# Then push:
docker push registry-sdc.argo.grnet.gr/myimage:20191231

SSL directory

# currently on Athens version:
mkdir /root/cert

Healthcheck directories

# currently on Athens version:
mkdir /root/HEALTH

Nginx health check is probably needed on every machines, so:

Download healthcheck file
Make it executable (I think by the user who runs docker, or by root might be enough)

# currently on Athens version:
cd /root/HEALTH
wget https://raw.githubusercontent.com/SeaDataCloud/vre-config/master/common/healthcheck_nginx.sh
# make it executable

Installing unison

Only on service nodes where data has to be synchronized to, i.e. nodes that cannot access the NextCloud data or directly or via NFS. [TODO]