How to run Ansible playbooks - usegalaxy-no/galaxyadmin GitHub Wiki

This page gives a general introduction on how to run the Ansible playbooks for the UseGalaxy.no infrastructure. For information on how to edit the playbooks or solve specific tasks, please confer one of the other pages in the FAQ. To follow these instructions, you must have Ansible installed already (not covered here) and also Git.

The playbooks should be executed on your own local machine and not directly on the UseGalaxy.no servers. Ansible will use SSH to login into the various servers, so in order to run the playbooks you will need to have "sysadmin" access to the servers [How to add sysadmins?]

First, make a clone of the infrastructure playbook to your local computer and change into this directory

git clone https://github.com/usegalaxy-no/infrastructure-playbook 
cd infrastructure-playbook

If you already have a clone of the repository, you can just run git pull in infrastructure-playbook directory the to get the latest changes.

Next, change into either the env/test or env/main directory depending on whether you want to run the playbooks on the test servers or the production servers.

cd env/test

When you run a playbook, Ansible will target the servers described in the hosts file in the same directory. The entries in the env/test/hosts file point to the test servers and the ones in the env/main/hosts file point to the production servers.

In order to run a playbook, you will also need to store the password to the Ansible vault in a file called vault_password in your current directory (e.g. env/test/vault_password). This is needed in order to decrypt secret values that are encrypted in the playbooks. If you don't have this vault password, you can obtain it from another administrator (or SSH into usegalaxy.no as sysadmin and find it in the file README_secrets.txt).

To run a playbook, you simply execute the command

ansible-playbook <playbook_file.yml>

There are several different playbooks, and which one to run will depend on which task you try to accomplish. Most playbooks are the same for prod and test, and the files in env/main and env/test are just symlinks to common files in env/common.

  • cvmfs-server.yml = Sets up the /cvmfs/data.usegalaxy.no repository service.
  • cvmfsproxy.yml = Sets up a cvmfs proxy service, and configures compute nodes to use this service.
  • database.yml = Sets up services (postgresql, mysql, rabbitmq, apache and postfix) on the database server.
  • ecc.yml = Sets up the eccd service on the head node (Galaxy web server).
  • galaxy.yml = Sets up Galaxy, and related services, on the main node.
  • jenkins.yml = Sets up Jenkins (not in use).
  • nga.yml = Sets up the nels-galaxy-api service.
  • pulsar.yml = Sets up Pulsar service (currently for testing, NVI).
  • system.yml = Configure linux system settings (i.e. time zone, fail2ban, disable ipv6) on all hosts.
  • workflows.yml = Install workflows from files/galaxy/workflows/ to Galaxy server.

However, some playbooks have separate copies for prod and test, or they are only available for one of the two:

  • slurm.yml = Sets up the Slurm queue system (differences between test and main should be merged, probably).
  • dns.yml = (This might not be needed anymore?)

If you run the galaxy.yml playbook (not to be confused with the group_vars/galaxy.yml configuration file), Ansible will restart the Galaxy server afterwards. This should not affect running jobs, but the Galaxy front-end will be unaccessible for a short time. Therefore you should only execute these playbooks at times when few or no users are using the service (at least if you target the production servers).

It is possible to run only specific parts of the playbook by adding tags to the command, e.g.:

ansible-playbook system.yml --tags "users"

Some recognized tags are:

  • firewall-setup = Only open firewall ports.
  • users = Only add users and groups.
  • telegraf = Only configure telegraf.

Troubleshooting

  • All the roles required by the playbooks should already be included in the repository, but if you get complaints about missing roles or undefined variables when running the playbook, you can try to reinstall them by running this command (assuming there is a file called requirements.yml in your current directory).
ansible-galaxy install -r requirements.yml
  • If you get complaints about missing modules when running a playbook, you may have to install some Python dependencies by running the following command in the infrastructure-playbook root directory. (You can create a virtual environment to install the modules in if you want.)
pip install -r requirements.txt