WMCore central services deployment k8 - dmwm/WMCore Wiki

This wiki describes all the steps related to the deployment of WMCore central services on a kubernetes cluster. The detailed documentation is provided by CMSWeb Team here: https://cms-http-group.docs.cern.ch/k8s_cluster/quickstart/

Setting up cluster environment:

CMSWeb original instructions

As explained in the document above before starting the actual service deployment One needs to setup the proper environment for the cluster where the deployment is about to happen. The possible clusters are as follows:

ssh lxplus8.cern.ch

workPath=`pwd`/deploymentK8
mkdir -p $workPath
cd $workPath
export OS_TOKEN=$(openstack token issue -c id -f value)

Fist one needs all cluster configuration files. They are now all uploaded in the users_config repository. NOTE: For access to the repository ask CMSWEB Team:

cd $workPath
git clone https://gitlab.cern.ch/cmsweb-k8s/users_config.git
  • Production:
export KUBECONFIG=$workPath/users_config/config.prod/config.cmsweb-k8s-services-prod
  • PreProduction/Testbed:
export KUBECONFIG=$workPath/users_config/config.preprod/config.cmsweb-testbed
  • Develop:
export OS_PROJECT_NAME="CMS Webtools Mig"
export KUBECONFIG=$workPath/users_config/config.test1/config.cmsweb-test1
# or:
export KUBECONFIG=$workPath/users_config/config.test5/config.cmsweb-test5
# or:
export KUBECONFIG=$workPath/users_config/config.test8/config.cmsweb-test8
# or:
export KUBECONFIG=$workPath/users_config/config.test9/config.cmsweb-test9
# or:
export KUBECONFIG=$workPath/users_config/config.test10/config.cmsweb-test10

Automating environment setup:

In order to automate the above steps, one may consider creating the following script (or even putting the whole k8init function from bellow in its ~/.bashrc):

NOTE: Please do change your workPath accordingly.

cat << 'EOF' > k8init.sh
#!/bin/bash


k8init(){
    workPath=~/WMCoreDev.d/deploy/deploymentK8
    [[ -d $workPath ]] || mkdir -p $workPath
    cd $workPath || { echo "ERROR: Missing working directory"; return ;}
    export OS_TOKEN=$(openstack token issue -c id -f value)
    case $1 in
        'prod')
            config=config.cmsweb-k8s-services-prod
            export KUBECONFIG=$workPath/users_config/config.prod/config.cmsweb-k8s-services-prod
            ;;
        'preprod')
            export KUBECONFIG=$workPath/users_config/config.preprod/config.cmsweb-testbed
            ;;
        'dev')
            export OS_PROJECT_NAME="CMS Webtools Mig"
            export KUBECONFIG=$workPath/users_config/config.test1/config.cmsweb-test1
            ;;
        'dev1')
            export OS_PROJECT_NAME="CMS Webtools Mig"
            export KUBECONFIG=$workPath/users_config/config.test1/config.cmsweb-test1
            ;;
        'dev5')
            export OS_PROJECT_NAME="CMS Webtools Mig"
            export KUBECONFIG=$workPath/users_config/config.test5/config.cmsweb-test5
            ;;
        'dev8')
            export OS_PROJECT_NAME="CMS Webtools Mig"
            export KUBECONFIG=$workPath/users_config/config.test8/config.cmsweb-test8
            ;;
        'dev9')
            export OS_PROJECT_NAME="CMS Webtools Mig"
            export KUBECONFIG=$workPath/users_config/config.test9/config.cmsweb-test9
            ;;
        'dev10')
            export OS_PROJECT_NAME="CMS Webtools Mig"
            export KUBECONFIG=$workPath/users_config/config.test10/config.cmsweb-test10
            ;;
        *)
            echo "Unsupported K8 cluster instance: $1"
            echo "You may chose between: [ 'prod' | 'preprod' | 'dev[1,5,8,9,10]*']"
            return 1
            ;;
    esac
}


k8init $*
EOF

chmod 755 k8init.sh

And then use it as:

./k8init.sh preprod

Optional:

To check if your environment have been set:

kubectl get pods -n dmwm

To login to a pod:

kubectl exec -it <pod-name> bash -n dmwm

To see logs of a pod:

kubectl logs <pod-name> -n dmwm

Secrets/config deployment:

CMSWeb original instructions

Before deploying any service one should check if all the needed configuration changes are merged in the service-config repository, using the table from bellow to map the current deployment to the proper branch in gitlab. Then:

  • Clone the service config repository and set it to the proper branch:

NOTE: You should clone it in a private area since the repository contains sensitive information! NOTE: The secrets themselves must be populated by hand (beware not to upload those in the gitlab repository itself). Here is how a local path with the secrets populated should look like:

── reqmgr2
│   ├── config.py
│   └── ReqMgr2Secrets.py
├── reqmgr2ms-monitor
│   ├── config-monitor.py
│   └── ReqMgr2MSSecrets.py
├── reqmgr2ms-output
│   ├── config-output.py
│   ├── manage
│   └── ReqMgr2MSSecrets.py
├── reqmgr2ms-rulecleaner
│   ├── config-ruleCleaner.py
│   └── ReqMgr2MSSecrets.py
├── reqmgr2ms-transferor
│   ├── config-transferor.py
│   └── ReqMgr2MSSecrets.py
├── reqmgr2ms-unmerged
│   ├── config-unmerged.py
│   └── ReqMgr2MSSecrets.py
├── reqmgr2-tasks
│   ├── config.py
│   └── ReqMgr2Secrets.py

And here are the commands to be executed:

git clone https://gitlab.cern.ch/cmsweb-k8s/services_config.git

cd services_config

git checkout preprod
git pull origin

cd ../
  • Clone the Kubernetes config repository:
git clone https://github.com/dmwm/CMSKubernetes.git
cd CMSKubernetes
git checkout master
git pull origin
git fetch origin
  • Go to the cmsweb area:
cd kubernetes/cmsweb
  • Populate the corresponding secrets where they belong as shown in the tree above.
  • Deploy secrets/configs:

The general format of the command is:

./scripts/deploy-secrets.sh <namespace> <service-name> <path_to_services_config>

Where:

  • <namespace> - should be dmwm
  • <service-name> - refers to the relevant K8 configuration file name taken form: CMSKubernetes/kubernetes/cmsweb/<service-name>.yaml, rather than to the docker image name for the service.
  • <path_to_services_config> - refers to the path where one have cloned the service config repository

e.g. Example for reqmgr2 service:

./scripts/deploy-secrets.sh dmwm reqmgr2 ../../../services_config

Service deployment:

CMSWeb original instructions

For service deployment one needs to clone the CMSKubernetes gitlab repository containing all the K8 specific configurations needed for depoloyng the different services.

  • Clone the Kubernetes config repository:
git clone https://github.com/dmwm/CMSKubernetes.git
cd CMSKubernetes
git checkout master
git pull origin
git fetch origin

  • Go to the cmsweb area:
cd kubernetes/cmsweb
  • Deploy the service:

The general format of the command is:

./scripts/deploy-srv.sh <service-name> <image-tag> <environment>

Where:

  • <service-name> - refers to the relevant K8 configuration file name taken form: CMSKubernetes/kubernetes/cmsweb/<service-name>.yaml, rather than to the docker image name for the service.

  • <image-tag> - refers to HG2***

  • <environment> - refers to the cluster name where the deployment should happen (rather than to the environment name itself), which maps to environment names as in the table bellow :

e.g. Example for testbed deployments:

./scripts/deploy-srv.sh reqmgr2 HG2106b preprod

NOTE: Immediately after deploying you may list the existing pods and you should see duplicated instances of pods per service:

  • The OLD pod in status Terminating
  • The NEW pod in status PodInitializing or Running

You'd better wait until pod instances equalize/normalize before continuing with any further debugging.

NOTE: In case you want to force things a little, then some pods may be deleted (especially while redeploying due to a config change):

kubectl delete pod ms-um-t2-t3-5db9fcf5b9-hppfv -n dmwm
kubectl get pods -n dmwm


NAME                               READY   STATUS              RESTARTS   AGE
reqmgr2-657c79f94-hdbvx            1/1     Terminating         0          3d
reqmgr2-657c79f94-pnbsl            1/1     Running             0          3d
reqmgr2-8c9dd6d8f-sd8bl            1/1     Running             0          20s
reqmgr2-8c9dd6d8f-vw6nx            1/1     PodInitializing     0          21s

Environment names mapping

 +-----------------+----------------+---------------------------+
 | EnvironmentName | ClusterName[s] | service-config BranchName |
 +-----------------+----------------+---------------------------+
 | prod            | prod           | prod                      |
 +-----------------+----------------+---------------------------+
 | testbed         | preprod        | preprod                   |
 +-----------------+----------------+---------------------------+
 | develop         | test[1-6]      | test                      |
 +-----------------+----------------+---------------------------+

The full set of central services deployment

For WMCore:

  • reqmgr2
  • reqmgr2-tasks
  • reqmgr2ms-monitor
  • reqmgr2ms-transferor
  • reqmgr2ms-ruleCleaner
  • reqmgr2ms-output
  • reqmgr2ms-unmerged-t1
  • reqmgr2ms-unmerged-t2t3us
  • reqmgr2ms-unmerged-t2t3
  • workqueue
  • reqmon
  • reqmon-tasks

NOTE: The following two services are still in VMs:

  • acdcserver
  • couchdb

For T0:

...

  • t0_reqmon

NOTE: This is to be discussed, since the T0 deployment is currently under a different name space tzero and we need to adapt it to the dmwm one:

kubectl get pods -n tzero

Automation:

So in order to automate the configuration and deployment of all WMCore central services one may use the following small set of commands (e.g. for Testbed):

ssh lxplus8.cern.ch

workPath=`pwd`/deploymentK8.`date -I`
mkdir $workPath
cd $workPath

git clone https://gitlab.cern.ch/cmsweb-k8s/users_config.git

export OS_TOKEN=$(openstack token issue -c id -f value)
export KUBECONFIG=$workPath/users_config/config.preprod/config.cmsweb-testbed

servicesConfig=$workPath/services_config
namespace=dmwm
envr=preprod
depTag=HG2111c
clusterName=preprod
serviceList="reqmgr2 \
             reqmgr2-tasks \
             reqmgr2ms-monitor \
             reqmgr2ms-transferor \
             reqmgr2ms-rulecleaner \
             reqmgr2ms-output \
             reqmgr2ms-unmerged-t1 \
             reqmgr2ms-unmerged-t2t3us \
             reqmgr2ms-unmerged-t2t3 \
             workqueue \
             reqmon \
             reqmon-tasks"

git clone https://gitlab.cern.ch/cmsweb-k8s/services_config.git

cd services_config
git checkout $envr
git pull origin
cd ../

git clone https://github.com/dmwm/CMSKubernetes.git
cd CMSKubernetes
git checkout master
git pull origin
git fetch origin

cd kubernetes/cmsweb

configErrList=""
deployErrList=""

for service in $serviceList
do
    ./scripts/deploy-secrets.sh $namespace $service $servicesConfig || configErrList="$configErrList $service"
    ./scripts/deploy-srv.sh $service $depTag $envr $clusterName || deployErrList="$deployErrList $service"
done

echo -e "Services failed configuration:  \n$configErrList"
echo -e "Services failed deployment:  \n$deployErrList"

sleep 5

kubectl get pods -n $namespace

Building Docker images

CMSWeb original instructions1

CMSWeb original instructions2

Obtaining CLI credentials for CERN Registry:

The whole process should be issued from an docker enabled VM:

ssh lxplus8.cern.ch
ssh <CHANGEME:yourVM>

sudo systemctl status docker
sudo systemctl start docker

workPath=`pwd`/deploymentK8
namespace=dmwm
envr=develop
depTag=HG2111c

cd $workPath

One needs to login at least once at CERN registry (The hub where all the images are uploaded) and obtain a CLI password for his user to be used from lxplus cluster VM. For the purpose login at: https://registry.cern.ch/harbor/projects/1771/repositories. Then: upper right corner Drop Down Box (username) -> User Profile -> CLI Secret (copy to clipboard button). In case you'd like to preserve it on a file it's better be encrypted. A suggested optional command could be:

echo "<CHANGEME: with string from clipboard>" | openssl enc -aes256 -out registry.pwd.enc

Login to CERN Registry from a VM or lxplus

docker login registry.cern.ch
Enter username
Enter CLI Secret

or:

docker login -u <CHANGEME:username> -p `openssl enc -aes256 -d -in registry.pwd.enc 2>/dev/null` registry.cern.ch

Build and upload the images

pkgList="reqmgr2 \
         reqmgr2ms \
         reqmgr2ms-unmerged \
         workqueue \
         reqmon \
         t0_reqmon"

cd $workPath/CMSKubernetes/
git checkout master
git pull upstream master
git fetch upstream
cd docker

CMSK8STAG=$depTag
CMSK8S=http://cmsweb-test1.cern.ch
CMSWEB_ENV=$envr

NOTE: Please pay attention that package names differ from service names during the deployment step!

For all cms packages (NOTE:please avoiod doing it for all):

./build.sh

Or if you want to build just the minimal set of images required for a full WMCore validation:

./build.sh $pkgList
⚠️ **GitHub.com Fallback** ⚠️