WMAgent in Docker - dmwm/WMCore GitHub Wiki

This page provides details of the initial assessment of running WMAgent in Docker. The Dockerfile for WMAgent can be found at https://github.com/dmwm/CMSKubernetes/tree/master/docker/wmagent. The standard "deploy-wmagent.sh" script was split into the "install.sh" and "run.sh" scripts used in the Dockerfile. This builds a single large image with all components baked in (MariaDB, CouchDB, etc.). Future work will split these services into separate containers.

Requires Docker to be installed an agent VM (vocmsXXXX) running a schedd. Alan can do this.

[user@vocmsXXXX ~]$ sudo yum install docker

Update the OPTIONS line in /etc/sysconfig/docker, adding -G dockerroot OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false -G dockerroot'

[user@vocmsXXXX ~]$ sudo systemctl start docker
[user@vocmsXXXX ~]$ sudo systemctl enable docker

Add users to the dockerroot group

[user@vocmsXXXX ~]$ sudo usermod -aG dockerroot <username>

Make sure you can run a docker command under your CERN account.

[user@vocmsXXXX ~]$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

If you get an error, double check you are in the dockerroot group and logout/login to make sure the new group membership is active.

Clone the CMSKubernetes repo

[user@vocmsXXXX ~]$ git clone https://github.com/dmwm/CMSKubernetes.git
[user@vocmsXXXX ~]$ cd cd CMSKubernetes/docker/wmagent/

These images have not been uploaded to any central Docker repo so you will have to build them yourself. Default build options are defined in install.sh. The image only contains things common to all agents. Set these to the tag numbers you want to build.

WMA_TAG=1.2.8
DEPLOY_TAG=HG1909e
WMA_ARCH=slc7_amd64_gcc630
REPO="comp=comp"

Run options are defined in run.sh. A JobSubmitter patch from PR 9453 is required if you want to actually run workflows. The run script configures things unique to an agent running in a container, initializes the agent config and databases.

WMA_TAG=1.2.8
DEPLOY_TAG=HG1909e
TEAMNAME=testbed-erik
CENTRAL_SERVICES=esg-dmwm-dev1.cern.ch
AG_NUM=0
FLAVOR=mysql
PATCHES="9453"

You have to edit both install.sh and run.sh before you build the image. WMA_TAG and DEPLOY_TAG in run.sh must match what is in install.sh

Building the image

docker build --network=host .

This will spit out an image ID which can be used to run the container. You can optionally specify an image tag in the build command.

You must bind mount several directories and update the selinux lables with the Z option.

• /data/certs
• /etc/condor (schedd runs on the host, not the container)
• /tmp
• /data/srv/wmagent/current/install (stateful service and component dirs)
• /data/srv/wmagent/current/config

You also need to bind mount the secrets file.

• /data/admin/wmagent/WMAgent.secrets

The Dockerfile is set to simply drop you to a login shell (instead of running run.sh automatically). This allows you to look around in the container and run run.sh manually when you want to initialize the agent. The install and config dirs will be initialized the first time you execute run.sh and a .dockerinit file will be placed to keep track of the initialization. Subsequent container restarts won't touch these directories.

Run command:

docker run --network=host --rm -h `hostname -f` -it \
-v /data/certs:/data/certs:Z \
-v /etc/condor:/etc/condor:Z \
-v /tmp:/tmp:Z \
-v /data/srv/wmagent/current/install:/data/srv/wmagent/current/install:Z \
-v /data/srv/wmagent/current/config:/data/srv/wmagent/current/config:Z \
-v /data/admin/wmagent/WMAgent.secrets:/data/admin/wmagent/WMAgent.secrets:Z \
<image>

Use the image id or tag you defined from when you built the container for in the run command.

At this point things should be very familiar. The run.sh script will have initialized MariaDB and CouchDB. Source the agent environment and $manage start-agent and your Docker agent should be up and running.